[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:04 dist-test-slave-24mn krb5kdc[12570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:04 dist-test-slave-24mn krb5kdc[12570](info): set up 2 sockets
Sep 26 14:50:04 dist-test-slave-24mn krb5kdc[12570](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 14:50:05.392251 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:50:05 dist-test-slave-24mn krb5kdc[12570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:50:05.500806 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:44697
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:32835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:44697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:05.602517 12583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:05.602748 12583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:05.602785 12583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:05.606747 12583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:05.606797 12583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:05.606815 12583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:05.606830 12583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:05.606845 12583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:05.611868 12583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:32835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:44697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:44697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:05.612792 12583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:05.613716 12583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:05.621874 12588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:05.621965 12583 server_base.cc:1034] running on GCE node
W20260926 14:50:05.621912 12589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:05.621902 12591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:05.622663 12583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:05.623662 12583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:05.624812 12583 hybrid_clock.cc:648] HybridClock initialized: now 1790434205624780 us; error 49 us; skew 500 ppm
Sep 26 14:50:05 dist-test-slave-24mn krb5kdc[12570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434205, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:05.627315 12583 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:50:05.628144 12583 webserver.cc:460] Webserver started at http://127.12.68.254:42165/ using document root <none> and password file <none>
I20260926 14:50:05.628624 12583 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:05.628679 12583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:05.628830 12583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:05.630707 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "e5ab83c02e224a9598b9515035221f34"
format_stamp: "Formatted at 2026-09-26 14:50:05 on dist-test-slave-24mn"
server_key: "b4d8d151e2339559a52a2318deeef5a9"
server_key_iv: "a84545e8e396fe4f8f7943dffda3e5b7"
server_key_version: "encryptionkey@0"
I20260926 14:50:05.631210 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "e5ab83c02e224a9598b9515035221f34"
format_stamp: "Formatted at 2026-09-26 14:50:05 on dist-test-slave-24mn"
server_key: "b4d8d151e2339559a52a2318deeef5a9"
server_key_iv: "a84545e8e396fe4f8f7943dffda3e5b7"
server_key_version: "encryptionkey@0"
I20260926 14:50:05.634500 12583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:50:05.636629 12598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:05.637719 12583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 14:50:05.637820 12583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "e5ab83c02e224a9598b9515035221f34"
format_stamp: "Formatted at 2026-09-26 14:50:05 on dist-test-slave-24mn"
server_key: "b4d8d151e2339559a52a2318deeef5a9"
server_key_iv: "a84545e8e396fe4f8f7943dffda3e5b7"
server_key_version: "encryptionkey@0"
I20260926 14:50:05.637912 12583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:05.649459 12583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:05.652190 12583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:05.652341 12583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:05.658598 12583 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:44697
I20260926 14:50:05.658609 12650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:44697 every 8 connection(s)
I20260926 14:50:05.659426 12583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:05.662000 12651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:05.665808 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 12583
I20260926 14:50:05.665979 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:05.667518 12563 external_mini_cluster.cc:1428] Setting key 9ef2fb7bc819bf738f000932f4c4df83
I20260926 14:50:05.667764 12651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34: Bootstrap starting.
I20260926 14:50:05.670560 12651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:05.671347 12651 log.cc:824] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:05.673034 12651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34: No bootstrap required, opened a new log
I20260926 14:50:05.676921 12651 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab83c02e224a9598b9515035221f34" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 44697 } }
I20260926 14:50:05.677168 12651 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:05.677248 12651 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ab83c02e224a9598b9515035221f34, State: Initialized, Role: FOLLOWER
I20260926 14:50:05.677685 12651 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab83c02e224a9598b9515035221f34" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 44697 } }
I20260926 14:50:05.677805 12651 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:05.677861 12651 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:05.677950 12651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:05.678733 12651 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab83c02e224a9598b9515035221f34" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 44697 } }
I20260926 14:50:05.679024 12651 leader_election.cc:302] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5ab83c02e224a9598b9515035221f34; no voters:
I20260926 14:50:05.679275 12651 leader_election.cc:288] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:05.679340 12656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:05.679528 12656 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [term 1 LEADER]: Becoming Leader. State: Replica: e5ab83c02e224a9598b9515035221f34, State: Running, Role: LEADER
I20260926 14:50:05.679796 12656 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab83c02e224a9598b9515035221f34" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 44697 } }
I20260926 14:50:05.680480 12651 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:05.681288 12657 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5ab83c02e224a9598b9515035221f34. Latest consensus state: current_term: 1 leader_uuid: "e5ab83c02e224a9598b9515035221f34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab83c02e224a9598b9515035221f34" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 44697 } } }
I20260926 14:50:05.681412 12657 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:05.681608 12657 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5ab83c02e224a9598b9515035221f34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab83c02e224a9598b9515035221f34" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 44697 } } }
I20260926 14:50:05.681687 12657 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:05.681957 12665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:05.684998 12665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 14:50:05 dist-test-slave-24mn krb5kdc[12570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:05.690027 12665 catalog_manager.cc:1347] Generated new cluster ID: d5fbf16829ea4c5bb1463d4eb090edd8
I20260926 14:50:05.690125 12665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:05.694557 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.670895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.12.68.254:44697)
0926 14:50:05.671339 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:05.671358 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:05.672433 (+ 1075us) server_negotiation.cc:407] Connection header received
0926 14:50:05.676503 (+ 4070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:05.676524 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:05.676889 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:05.677235 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:05.679439 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.680595 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.682209 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.682931 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.683047 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.687912 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:05.687919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:05.687933 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:05.687961 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:05.690037 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:05.690668 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:05.690674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:05.690679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:05.690812 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:05.691190 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:05.691193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:05.691195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:05.691682 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:05.691863 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:05.693439 (+ 1576us) server_negotiation.cc:299] Negotiation successful
0926 14:50:05.693706 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":122,"threads_started":1}
I20260926 14:50:05.696969 12665 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:05.697755 12665 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:05.704227 12665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5ab83c02e224a9598b9515035221f34: Generated new TSK 0
I20260926 14:50:05.704882 12665 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:50:05.728633 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:44697
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:32835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:05.832585 12677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:05.832808 12677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:05.832854 12677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:05.836714 12677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:05.836761 12677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:05.836831 12677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:05.841821 12677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:32835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:44697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:05.842759 12677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:05.843683 12677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:05.852361 12682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:05.852407 12685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:05.852384 12683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:05.852560 12677 server_base.cc:1034] running on GCE node
I20260926 14:50:05.852950 12677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:05.853518 12677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:05.854666 12677 hybrid_clock.cc:648] HybridClock initialized: now 1790434205854649 us; error 33 us; skew 500 ppm
Sep 26 14:50:05 dist-test-slave-24mn krb5kdc[12570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434205, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:05.857136 12677 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:50:05.857928 12677 webserver.cc:460] Webserver started at http://127.12.68.193:43403/ using document root <none> and password file <none>
I20260926 14:50:05.858415 12677 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:05.858474 12677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:05.858664 12677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:05.860541 12677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "f22577dbbb8340f2887c7347fb972557"
format_stamp: "Formatted at 2026-09-26 14:50:05 on dist-test-slave-24mn"
server_key: "d8b0b410a7e9f2e890dd1a42fc5f2508"
server_key_iv: "a671da0518d3088f95e85780e4652af8"
server_key_version: "encryptionkey@0"
I20260926 14:50:05.861037 12677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "f22577dbbb8340f2887c7347fb972557"
format_stamp: "Formatted at 2026-09-26 14:50:05 on dist-test-slave-24mn"
server_key: "d8b0b410a7e9f2e890dd1a42fc5f2508"
server_key_iv: "a671da0518d3088f95e85780e4652af8"
server_key_version: "encryptionkey@0"
I20260926 14:50:05.864526 12677 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:05.866617 12692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:05.867708 12677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:05.867808 12677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "f22577dbbb8340f2887c7347fb972557"
format_stamp: "Formatted at 2026-09-26 14:50:05 on dist-test-slave-24mn"
server_key: "d8b0b410a7e9f2e890dd1a42fc5f2508"
server_key_iv: "a671da0518d3088f95e85780e4652af8"
server_key_version: "encryptionkey@0"
I20260926 14:50:05.867915 12677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:05.886008 12677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:05.889221 12677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:05.889405 12677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:05.889920 12677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:05.890663 12677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:05.890707 12677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:05.890756 12677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:05.890785 12677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:05.899107 12677 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:36917
I20260926 14:50:05.899132 12805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:36917 every 8 connection(s)
I20260926 14:50:05.899938 12677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:05.903469 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 12677
I20260926 14:50:05.903570 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:05.903812 12563 external_mini_cluster.cc:1428] Setting key f29a9e3a8dc3d8c2baf73068d6750f22
Sep 26 14:50:05 dist-test-slave-24mn krb5kdc[12570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434205, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
I20260926 14:50:05.912243 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.901555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:43901 (local address 127.12.68.254:44697)
0926 14:50:05.901671 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:05.901675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:05.902608 (+ 933us) server_negotiation.cc:407] Connection header received
0926 14:50:05.903414 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:05.903418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:05.903482 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:05.903559 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:05.904829 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.905420 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.906484 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.906711 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.906795 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.909178 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:05.909181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:05.909183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:05.909204 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:05.910445 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:05.910886 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:05.910889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:05.910890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:05.910961 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:05.911244 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:05.911248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:05.911249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:05.911408 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:05.911491 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:05.911971 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 14:50:05.912089 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:50:05.913014 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.901871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:44697 (local address 127.12.68.193:43901)
0926 14:50:05.902475 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:05.902514 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:05.903222 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:05.903785 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:05.903792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:05.904172 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:05.904691 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:05.904706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.905533 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.905536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:05.906367 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:05.906375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.906834 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.906838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:05.906995 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.907520 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:05.907534 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:05.909033 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:05.910553 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:05.910558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:05.910571 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:05.910799 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:05.911038 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:05.911041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:05.911043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:05.911164 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:05.911509 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:05.911515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:05.911725 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 14:50:05.911928 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 14:50:05.912221 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
I20260926 14:50:05.914121 12806 heartbeater.cc:347] Connected to a master server at 127.12.68.254:44697
I20260926 14:50:05.914326 12806 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:05.914739 12806 heartbeater.cc:511] Master 127.12.68.254:44697 requested a full tablet report, sending...
I20260926 14:50:05.916229 12614 ts_manager.cc:194] Registered new tserver with Master: f22577dbbb8340f2887c7347fb972557 (127.12.68.193:36917)
I20260926 14:50:05.917266 12614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:43901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:50:05.926858 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:44697
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:32835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:06.029045 12811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:06.029281 12811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:06.029322 12811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:06.033210 12811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:06.033255 12811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:06.033315 12811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:06.038205 12811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:32835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:44697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:06.039067 12811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:06.039978 12811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:06.048627 12816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:06.048748 12811 server_base.cc:1034] running on GCE node
W20260926 14:50:06.048641 12817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:06.048641 12819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:06.049456 12811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:06.049988 12811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:06.051120 12811 hybrid_clock.cc:648] HybridClock initialized: now 1790434206051105 us; error 27 us; skew 500 ppm
Sep 26 14:50:06 dist-test-slave-24mn krb5kdc[12570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434206, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:06.053535 12811 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:50:06.054272 12811 webserver.cc:460] Webserver started at http://127.12.68.194:43065/ using document root <none> and password file <none>
I20260926 14:50:06.054749 12811 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:06.054807 12811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:06.054992 12811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:06.056898 12811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "ae8e458d236c487daf165962e2c4cd86"
format_stamp: "Formatted at 2026-09-26 14:50:06 on dist-test-slave-24mn"
server_key: "9fe6f92435ffe793cf166b1c2d772bf9"
server_key_iv: "8b8e015c5081e00b6eb3551a2cb12566"
server_key_version: "encryptionkey@0"
I20260926 14:50:06.057395 12811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "ae8e458d236c487daf165962e2c4cd86"
format_stamp: "Formatted at 2026-09-26 14:50:06 on dist-test-slave-24mn"
server_key: "9fe6f92435ffe793cf166b1c2d772bf9"
server_key_iv: "8b8e015c5081e00b6eb3551a2cb12566"
server_key_version: "encryptionkey@0"
I20260926 14:50:06.060928 12811 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 14:50:06.063131 12826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:06.064286 12811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:06.064393 12811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "ae8e458d236c487daf165962e2c4cd86"
format_stamp: "Formatted at 2026-09-26 14:50:06 on dist-test-slave-24mn"
server_key: "9fe6f92435ffe793cf166b1c2d772bf9"
server_key_iv: "8b8e015c5081e00b6eb3551a2cb12566"
server_key_version: "encryptionkey@0"
I20260926 14:50:06.064488 12811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:06.083943 12811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:06.086337 12811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:06.086488 12811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:06.086985 12811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:06.087752 12811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:06.087797 12811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:06.087836 12811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:06.087864 12811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:06.095896 12811 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:45353
I20260926 14:50:06.095968 12939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:45353 every 8 connection(s)
I20260926 14:50:06.096694 12811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:06.101804 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 12811
I20260926 14:50:06.101902 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:06.102170 12563 external_mini_cluster.cc:1428] Setting key b5ccd30e1fd5cdb9e53c4136075d01d3
Sep 26 14:50:06 dist-test-slave-24mn krb5kdc[12570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434206, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:06.109503 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:06.098252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:47543 (local address 127.12.68.254:44697)
0926 14:50:06.098392 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:06.098397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:06.099369 (+ 972us) server_negotiation.cc:407] Connection header received
0926 14:50:06.100149 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:06.100151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:06.100193 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:06.100270 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:06.101264 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.101795 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.103201 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.103499 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.103586 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:06.106238 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:06.106242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:06.106244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:06.106265 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:06.107527 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:06.108109 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:06.108112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:06.108113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:06.108183 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:06.108517 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:06.108519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:06.108520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:06.108669 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:06.108735 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:06.109283 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 14:50:06.109392 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:50:06.110527 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:06.098645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:44697 (local address 127.12.68.194:47543)
0926 14:50:06.099231 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:06.099268 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:06.099987 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:06.100419 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:06.100425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:06.100730 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:06.101151 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:06.101162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.101926 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.101930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:06.103086 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:06.103095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.103642 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.103646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:06.103827 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:06.104350 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:06.104367 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:06.106086 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:06.107634 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:06.107640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:06.107655 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:06.108019 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:06.108260 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:06.108262 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:06.108264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:06.108437 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:06.108754 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:06.108761 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:06.109041 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 14:50:06.109257 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 14:50:06.109574 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
I20260926 14:50:06.111575 12940 heartbeater.cc:347] Connected to a master server at 127.12.68.254:44697
I20260926 14:50:06.111768 12940 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:06.112202 12940 heartbeater.cc:511] Master 127.12.68.254:44697 requested a full tablet report, sending...
I20260926 14:50:06.113119 12614 ts_manager.cc:194] Registered new tserver with Master: ae8e458d236c487daf165962e2c4cd86 (127.12.68.194:45353)
I20260926 14:50:06.113639 12614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:47543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:50:06.126709 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:44697
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:32835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:06.230264 12945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:06.230489 12945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:06.230530 12945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:06.234382 12945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:06.234431 12945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:06.234485 12945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:06.239363 12945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:32835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:44697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:06.240240 12945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:06.241174 12945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:06.249820 12951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:06.249939 12945 server_base.cc:1034] running on GCE node
W20260926 14:50:06.249847 12950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:06.249886 12953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:06.250442 12945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:06.251001 12945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:06.252148 12945 hybrid_clock.cc:648] HybridClock initialized: now 1790434206252127 us; error 32 us; skew 500 ppm
Sep 26 14:50:06 dist-test-slave-24mn krb5kdc[12570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434206, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:06.254606 12945 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:50:06.255357 12945 webserver.cc:460] Webserver started at http://127.12.68.195:39037/ using document root <none> and password file <none>
I20260926 14:50:06.255867 12945 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:06.255964 12945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:06.256148 12945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:06.258019 12945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "4a2a1b75dbb84f8990cb551aff0a5dfc"
format_stamp: "Formatted at 2026-09-26 14:50:06 on dist-test-slave-24mn"
server_key: "6dca707f9cffc2347ca9e622296bb321"
server_key_iv: "e2e85fbe2fed0d4c56dfac63f9bd123e"
server_key_version: "encryptionkey@0"
I20260926 14:50:06.258517 12945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "4a2a1b75dbb84f8990cb551aff0a5dfc"
format_stamp: "Formatted at 2026-09-26 14:50:06 on dist-test-slave-24mn"
server_key: "6dca707f9cffc2347ca9e622296bb321"
server_key_iv: "e2e85fbe2fed0d4c56dfac63f9bd123e"
server_key_version: "encryptionkey@0"
I20260926 14:50:06.261862 12945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 14:50:06.264178 12960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:06.265352 12945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:50:06.265460 12945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "4a2a1b75dbb84f8990cb551aff0a5dfc"
format_stamp: "Formatted at 2026-09-26 14:50:06 on dist-test-slave-24mn"
server_key: "6dca707f9cffc2347ca9e622296bb321"
server_key_iv: "e2e85fbe2fed0d4c56dfac63f9bd123e"
server_key_version: "encryptionkey@0"
I20260926 14:50:06.265547 12945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:06.283921 12945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:06.286322 12945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:06.286471 12945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:06.286958 12945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:06.287674 12945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:06.287719 12945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:06.287765 12945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:06.287793 12945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:06.295794 12945 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:37309
I20260926 14:50:06.295830 13073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:37309 every 8 connection(s)
I20260926 14:50:06.296578 12945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:06.301738 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 12945
I20260926 14:50:06.301843 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:06.302088 12563 external_mini_cluster.cc:1428] Setting key 47e05a55b6d5e81e5683cc080341990b
Sep 26 14:50:06 dist-test-slave-24mn krb5kdc[12570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434206, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:06.308740 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:06.298110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:55339 (local address 127.12.68.254:44697)
0926 14:50:06.298265 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:06.298270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:06.299173 (+ 903us) server_negotiation.cc:407] Connection header received
0926 14:50:06.299987 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:06.299990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:06.300033 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:06.300101 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:06.301118 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.301636 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.303002 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.303215 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.303299 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:06.305746 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:06.305749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:06.305751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:06.305771 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:06.306909 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:06.307351 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:06.307353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:06.307355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:06.307424 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:06.307765 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:06.307768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:06.307770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:06.307984 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:06.308067 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:06.308484 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 14:50:06.308591 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 14:50:06.309413 13076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:06.298460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:44697 (local address 127.12.68.195:55339)
0926 14:50:06.299034 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:06.299070 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:06.299788 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:06.300266 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:06.300272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:06.300586 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:06.301011 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:06.301023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.301769 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.301772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:06.302885 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:06.302894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.303342 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.303346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:06.303499 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:06.303964 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:06.303979 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:06.305604 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:06.307005 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:06.307010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:06.307023 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:06.307240 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:06.307502 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:06.307505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:06.307507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:06.307676 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:06.308071 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:06.308077 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:06.308277 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 14:50:06.308428 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 14:50:06.308700 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":99,"threads_started":1}
I20260926 14:50:06.310364 13074 heartbeater.cc:347] Connected to a master server at 127.12.68.254:44697
I20260926 14:50:06.310550 13074 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:06.310942 13074 heartbeater.cc:511] Master 127.12.68.254:44697 requested a full tablet report, sending...
I20260926 14:50:06.311829 12614 ts_manager.cc:194] Registered new tserver with Master: 4a2a1b75dbb84f8990cb551aff0a5dfc (127.12.68.195:37309)
I20260926 14:50:06.312398 12614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:55339
I20260926 14:50:06.315753 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 14:50:06 dist-test-slave-24mn krb5kdc[12570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434206, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:50:06 dist-test-slave-24mn krb5kdc[12570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434206, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.68.193@KRBTEST.COM
I20260926 14:50:06.338989 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:06.330391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.12.68.193:36917)
0926 14:50:06.330701 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:06.330712 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:06.330736 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:50:06.330794 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:06.330800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:06.330964 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:06.331080 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:06.331700 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.332483 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.333265 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.333611 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.333699 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:06.335958 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:06.335966 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:06.335977 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:06.336009 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:06.337419 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:06.337827 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:06.337832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:06.337833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:06.337925 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:06.338144 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:06.338147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:06.338148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:06.338388 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:06.338528 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:06.338732 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 14:50:06.338851 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":118,"threads_started":1}
W20260926 14:50:06.339902 12720 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46352
Sep 26 14:50:06 dist-test-slave-24mn krb5kdc[12570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:50:06 dist-test-slave-24mn krb5kdc[12570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.193@KRBTEST.COM
I20260926 14:50:06.362517 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:06.355431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.12.68.193:36917)
0926 14:50:06.355565 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:06.355570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:06.355694 (+ 124us) server_negotiation.cc:407] Connection header received
0926 14:50:06.355842 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:06.355845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:06.355905 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:06.355975 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:06.356520 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.357010 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.357762 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.358000 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.358080 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:06.360109 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:06.360112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:06.360114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:06.360135 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:06.361338 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:06.361659 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:06.361661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:06.361662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:06.361722 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:06.361897 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:06.361898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:06.361899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:06.362033 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:06.362119 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:06.362264 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 14:50:06.362367 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:50:06.363394 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 12677
I20260926 14:50:06.367996 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 12811
I20260926 14:50:06.372314 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 12945
I20260926 14:50:06.376278 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 12583
2026-09-26T14:50:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1945 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:06 dist-test-slave-24mn krb5kdc[13088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:06 dist-test-slave-24mn krb5kdc[13088](info): set up 2 sockets
Sep 26 14:50:06 dist-test-slave-24mn krb5kdc[13088](info): commencing operation
krb5kdc: starting...
W20260926 14:50:07.319291 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:50:07 dist-test-slave-24mn krb5kdc[13088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:50:07.429121 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:42115
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:34577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:42115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:07.533793 13101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:07.534054 13101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:07.534108 13101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:07.538046 13101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:07.538102 13101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:07.538128 13101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:07.538148 13101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:07.538169 13101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:07.543345 13101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:34577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:42115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:42115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:07.544404 13101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:07.545415 13101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:07.553313 13106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:07.553283 13109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:07.553321 13107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:07.553359 13101 server_base.cc:1034] running on GCE node
I20260926 14:50:07.554090 13101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:07.555135 13101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:07.556294 13101 hybrid_clock.cc:648] HybridClock initialized: now 1790434207556262 us; error 43 us; skew 500 ppm
Sep 26 14:50:07 dist-test-slave-24mn krb5kdc[13088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434207, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:07.558808 13101 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:50:07.559636 13101 webserver.cc:460] Webserver started at http://127.12.68.254:45751/ using document root <none> and password file <none>
I20260926 14:50:07.560194 13101 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:07.560258 13101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:07.560462 13101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:07.562420 13101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "f424a55b604a46adb163e5e056b59335"
format_stamp: "Formatted at 2026-09-26 14:50:07 on dist-test-slave-24mn"
server_key: "404520b2629c8495b338ed2f23a332ea"
server_key_iv: "6cca7eb1f08c29df1ae7f7508c8214ed"
server_key_version: "encryptionkey@0"
I20260926 14:50:07.562943 13101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "f424a55b604a46adb163e5e056b59335"
format_stamp: "Formatted at 2026-09-26 14:50:07 on dist-test-slave-24mn"
server_key: "404520b2629c8495b338ed2f23a332ea"
server_key_iv: "6cca7eb1f08c29df1ae7f7508c8214ed"
server_key_version: "encryptionkey@0"
I20260926 14:50:07.566434 13101 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:50:07.568687 13116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:07.569850 13101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:07.569959 13101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "f424a55b604a46adb163e5e056b59335"
format_stamp: "Formatted at 2026-09-26 14:50:07 on dist-test-slave-24mn"
server_key: "404520b2629c8495b338ed2f23a332ea"
server_key_iv: "6cca7eb1f08c29df1ae7f7508c8214ed"
server_key_version: "encryptionkey@0"
I20260926 14:50:07.570050 13101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:07.583175 13101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:07.585961 13101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:07.586128 13101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:07.592638 13101 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:42115
I20260926 14:50:07.592659 13168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:42115 every 8 connection(s)
I20260926 14:50:07.593573 13101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:07.594450 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 13101
I20260926 14:50:07.594559 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:07.594828 12563 external_mini_cluster.cc:1428] Setting key 6a6f0a9848b6aebf9912c705098918c0
I20260926 14:50:07.596720 13169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:07.602695 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335: Bootstrap starting.
Sep 26 14:50:07 dist-test-slave-24mn krb5kdc[13088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:07.604948 13169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:07.605605 13169 log.cc:824] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:07.607327 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335: No bootstrap required, opened a new log
I20260926 14:50:07.608212 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.596156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.12.68.254:42115)
0926 14:50:07.596538 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.596553 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.596588 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:50:07.597272 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.597289 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.597613 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.597895 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:07.598695 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.599686 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.600842 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.601300 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.601421 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.604085 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:07.604090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:07.604102 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:07.604129 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:07.605623 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.606027 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.606031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.606035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.606136 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.606431 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.606433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.606434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.606797 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:07.606985 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.607250 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.607501 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
I20260926 14:50:07.611032 13169 raft_consensus.cc:348] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f424a55b604a46adb163e5e056b59335" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42115 } }
I20260926 14:50:07.611239 13169 raft_consensus.cc:374] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:07.611302 13169 raft_consensus.cc:729] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f424a55b604a46adb163e5e056b59335, State: Initialized, Role: FOLLOWER
I20260926 14:50:07.611668 13169 consensus_queue.cc:260] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f424a55b604a46adb163e5e056b59335" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42115 } }
I20260926 14:50:07.611786 13169 raft_consensus.cc:388] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:07.611847 13169 raft_consensus.cc:482] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:07.611967 13169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:07.612782 13169 raft_consensus.cc:504] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f424a55b604a46adb163e5e056b59335" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42115 } }
I20260926 14:50:07.613076 13169 leader_election.cc:302] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f424a55b604a46adb163e5e056b59335; no voters:
I20260926 14:50:07.613354 13169 leader_election.cc:288] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:07.613464 13174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:07.613677 13174 raft_consensus.cc:686] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [term 1 LEADER]: Becoming Leader. State: Replica: f424a55b604a46adb163e5e056b59335, State: Running, Role: LEADER
I20260926 14:50:07.614010 13174 consensus_queue.cc:237] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f424a55b604a46adb163e5e056b59335" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42115 } }
I20260926 14:50:07.614497 13169 sys_catalog.cc:564] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:07.615729 13175 sys_catalog.cc:455] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f424a55b604a46adb163e5e056b59335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f424a55b604a46adb163e5e056b59335" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42115 } } }
I20260926 14:50:07.615738 13176 sys_catalog.cc:455] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f424a55b604a46adb163e5e056b59335. Latest consensus state: current_term: 1 leader_uuid: "f424a55b604a46adb163e5e056b59335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f424a55b604a46adb163e5e056b59335" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42115 } } }
I20260926 14:50:07.615988 13176 sys_catalog.cc:458] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:07.615923 13175 sys_catalog.cc:458] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335 [sys.catalog]: This master's current role is: LEADER
W20260926 14:50:07.618944 13189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:50:07.619004 13189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:50:07.619076 13187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:07.619726 13187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:07.624922 13187 catalog_manager.cc:1347] Generated new cluster ID: dac7e4b2d0bc46989c398395c4ed9198
I20260926 14:50:07.624981 13187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:07.641924 13187 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:07.643169 13187 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:07.652362 13187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f424a55b604a46adb163e5e056b59335: Generated new TSK 0
I20260926 14:50:07.653152 13187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:50:07.682427 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42115
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:34577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:07.788285 13195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:07.788517 13195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:07.788558 13195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:07.792536 13195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:07.792593 13195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:07.792660 13195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:07.797827 13195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:34577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:07.798790 13195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:07.799757 13195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:07.808676 13203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:07.808697 13200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:07.808758 13201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:07.808949 13195 server_base.cc:1034] running on GCE node
I20260926 14:50:07.809319 13195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:07.809882 13195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:07.811029 13195 hybrid_clock.cc:648] HybridClock initialized: now 1790434207811000 us; error 42 us; skew 500 ppm
Sep 26 14:50:07 dist-test-slave-24mn krb5kdc[13088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434207, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:07.813578 13195 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:50:07.814347 13195 webserver.cc:460] Webserver started at http://127.12.68.193:45301/ using document root <none> and password file <none>
I20260926 14:50:07.814930 13195 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:07.814994 13195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:07.815167 13195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:07.817162 13195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "ee0642ecccb5489d8d3cd19b4685bf5b"
format_stamp: "Formatted at 2026-09-26 14:50:07 on dist-test-slave-24mn"
server_key: "e80ee11de7377a0fcd2dcda5d074e704"
server_key_iv: "0e6764db4d945f6839d7c344d8d636f0"
server_key_version: "encryptionkey@0"
I20260926 14:50:07.817703 13195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "ee0642ecccb5489d8d3cd19b4685bf5b"
format_stamp: "Formatted at 2026-09-26 14:50:07 on dist-test-slave-24mn"
server_key: "e80ee11de7377a0fcd2dcda5d074e704"
server_key_iv: "0e6764db4d945f6839d7c344d8d636f0"
server_key_version: "encryptionkey@0"
I20260926 14:50:07.821180 13195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:07.823387 13210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:07.824555 13195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:07.824652 13195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "ee0642ecccb5489d8d3cd19b4685bf5b"
format_stamp: "Formatted at 2026-09-26 14:50:07 on dist-test-slave-24mn"
server_key: "e80ee11de7377a0fcd2dcda5d074e704"
server_key_iv: "0e6764db4d945f6839d7c344d8d636f0"
server_key_version: "encryptionkey@0"
I20260926 14:50:07.824740 13195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:07.839030 13195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:07.841691 13195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:07.841858 13195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:07.842381 13195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:07.843138 13195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:07.843184 13195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:07.843232 13195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:07.843261 13195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:07.850947 13195 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:34753
I20260926 14:50:07.850994 13323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:34753 every 8 connection(s)
I20260926 14:50:07.851904 13195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:07.858165 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 13195
I20260926 14:50:07.858253 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:07.858510 12563 external_mini_cluster.cc:1428] Setting key c224cb37cd1d5025e707e78ffa5ecd2e
Sep 26 14:50:07 dist-test-slave-24mn krb5kdc[13088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434207, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:07.864910 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.853623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:36527 (local address 127.12.68.254:42115)
0926 14:50:07.853747 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.853751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.854535 (+ 784us) server_negotiation.cc:407] Connection header received
0926 14:50:07.855741 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.855745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.855800 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.855869 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:07.857003 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.857539 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.858700 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.858980 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.859077 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.861549 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:07.861553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:07.861555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:07.861584 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:07.862947 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.863398 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.863401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.863404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.863495 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.863830 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.863833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.863835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.864079 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:07.864176 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.864619 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.864749 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:50:07.865589 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.853982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42115 (local address 127.12.68.193:36527)
0926 14:50:07.854396 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.854435 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.855540 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.856065 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.856071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.856399 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:07.856857 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.856871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.857651 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.857654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.858589 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.858599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.859116 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.859120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.859280 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.859792 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:07.859807 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:07.861388 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:07.863050 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:07.863055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:07.863065 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:07.863304 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:07.863576 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:07.863579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:07.863580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:07.863738 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:07.864169 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:07.864174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:07.864398 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.864545 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.864796 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260926 14:50:07.866747 13324 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42115
I20260926 14:50:07.866994 13324 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
I20260926 14:50:07.867544 13324 heartbeater.cc:511] Master 127.12.68.254:42115 requested a full tablet report, sending...
I20260926 14:50:07.868979 13133 ts_manager.cc:194] Registered new tserver with Master: ee0642ecccb5489d8d3cd19b4685bf5b (127.12.68.193:34753)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
I20260926 14:50:07.870090 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:36527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:50:07.882138 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42115
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:34577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:07.986999 13329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:07.987236 13329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:07.987277 13329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:07.991235 13329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:07.991288 13329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:07.991349 13329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:07.996413 13329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:34577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:07.997385 13329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:07.998333 13329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:08.007329 13337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:08.007422 13329 server_base.cc:1034] running on GCE node
W20260926 14:50:08.007329 13334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:08.007375 13335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:08.008104 13329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:08.008728 13329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:08.009886 13329 hybrid_clock.cc:648] HybridClock initialized: now 1790434208009866 us; error 34 us; skew 500 ppm
Sep 26 14:50:08 dist-test-slave-24mn krb5kdc[13088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434208, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:08.012465 13329 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:50:08.013345 13329 webserver.cc:460] Webserver started at http://127.12.68.194:34253/ using document root <none> and password file <none>
I20260926 14:50:08.013875 13329 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:08.013936 13329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:08.014142 13329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:08.016227 13329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "7436ab04f7af4cde8cefd904be89cc5c"
format_stamp: "Formatted at 2026-09-26 14:50:08 on dist-test-slave-24mn"
server_key: "190c2f0755c162a5f76de55d55bc924c"
server_key_iv: "cf307cb144adfe049e631eeaff898c63"
server_key_version: "encryptionkey@0"
I20260926 14:50:08.016748 13329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "7436ab04f7af4cde8cefd904be89cc5c"
format_stamp: "Formatted at 2026-09-26 14:50:08 on dist-test-slave-24mn"
server_key: "190c2f0755c162a5f76de55d55bc924c"
server_key_iv: "cf307cb144adfe049e631eeaff898c63"
server_key_version: "encryptionkey@0"
I20260926 14:50:08.020315 13329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:08.022677 13344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:08.023927 13329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:08.024036 13329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "7436ab04f7af4cde8cefd904be89cc5c"
format_stamp: "Formatted at 2026-09-26 14:50:08 on dist-test-slave-24mn"
server_key: "190c2f0755c162a5f76de55d55bc924c"
server_key_iv: "cf307cb144adfe049e631eeaff898c63"
server_key_version: "encryptionkey@0"
I20260926 14:50:08.024125 13329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:08.032871 13329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:08.035507 13329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:08.035665 13329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:08.036244 13329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:08.037005 13329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:08.037050 13329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:08.037099 13329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:08.037128 13329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:08.044932 13329 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:43149
I20260926 14:50:08.044957 13457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:43149 every 8 connection(s)
I20260926 14:50:08.045745 13329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:08.047308 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 13329
I20260926 14:50:08.047420 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:08.047680 12563 external_mini_cluster.cc:1428] Setting key 3326052d7feb488fdd47cf777f96b866
Sep 26 14:50:08 dist-test-slave-24mn krb5kdc[13088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434208, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
I20260926 14:50:08.059041 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.047477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:50991 (local address 127.12.68.254:42115)
0926 14:50:08.047612 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:08.047616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:08.048580 (+ 964us) server_negotiation.cc:407] Connection header received
0926 14:50:08.049940 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:08.049943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:08.049994 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:08.050064 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:08.051172 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.051684 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.052905 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.053119 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.053202 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.055733 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:08.055736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:08.055738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:08.055762 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:08.057148 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:08.057600 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:08.057603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:08.057605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:08.057689 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:08.057955 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:08.057958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:08.057959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:08.058159 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:08.058235 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:08.058757 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 14:50:08.058880 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:50:08.059736 13460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.047935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42115 (local address 127.12.68.194:50991)
0926 14:50:08.048430 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:08.048484 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:08.049745 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:08.050248 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:08.050254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:08.050590 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:08.051040 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:08.051052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.051801 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.051805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:08.052794 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:08.052803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.053244 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.053248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:08.053417 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.053918 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:08.053933 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:08.055566 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:08.057253 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:08.057258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:08.057269 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:08.057519 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:08.057767 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:08.057769 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:08.057770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:08.057894 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:08.058250 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:08.058256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:08.058516 (+ 260us) client_negotiation.cc:769] Sending connection context
0926 14:50:08.058670 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 14:50:08.058909 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":110,"threads_started":1}
I20260926 14:50:08.060882 13458 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42115
I20260926 14:50:08.061115 13458 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:08.061607 13458 heartbeater.cc:511] Master 127.12.68.254:42115 requested a full tablet report, sending...
I20260926 14:50:08.062641 13133 ts_manager.cc:194] Registered new tserver with Master: 7436ab04f7af4cde8cefd904be89cc5c (127.12.68.194:43149)
I20260926 14:50:08.063431 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:50991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:50:08.072664 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42115
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:34577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:08.177613 13463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:08.177868 13463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:08.177929 13463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:08.181890 13463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:08.181947 13463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:08.182027 13463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:08.187093 13463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:34577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:08.188113 13463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:08.189069 13463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:08.198276 13468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:08.198387 13463 server_base.cc:1034] running on GCE node
W20260926 14:50:08.198308 13471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:08.198269 13469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:08.198782 13463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:08.199381 13463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:08.200532 13463 hybrid_clock.cc:648] HybridClock initialized: now 1790434208200514 us; error 34 us; skew 500 ppm
Sep 26 14:50:08 dist-test-slave-24mn krb5kdc[13088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434208, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:08.203127 13463 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:50:08.204085 13463 webserver.cc:460] Webserver started at http://127.12.68.195:46303/ using document root <none> and password file <none>
I20260926 14:50:08.204622 13463 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:08.204689 13463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:08.204869 13463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:08.206806 13463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "185827d21319457495ba3bc80fdc181d"
format_stamp: "Formatted at 2026-09-26 14:50:08 on dist-test-slave-24mn"
server_key: "07342fd107551a6132fa4e2e95c81f06"
server_key_iv: "db32968efc96dcf080024da134d8d3ee"
server_key_version: "encryptionkey@0"
I20260926 14:50:08.207340 13463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "185827d21319457495ba3bc80fdc181d"
format_stamp: "Formatted at 2026-09-26 14:50:08 on dist-test-slave-24mn"
server_key: "07342fd107551a6132fa4e2e95c81f06"
server_key_iv: "db32968efc96dcf080024da134d8d3ee"
server_key_version: "encryptionkey@0"
I20260926 14:50:08.210880 13463 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:50:08.213265 13478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:08.214552 13463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:08.214666 13463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "185827d21319457495ba3bc80fdc181d"
format_stamp: "Formatted at 2026-09-26 14:50:08 on dist-test-slave-24mn"
server_key: "07342fd107551a6132fa4e2e95c81f06"
server_key_iv: "db32968efc96dcf080024da134d8d3ee"
server_key_version: "encryptionkey@0"
I20260926 14:50:08.214777 13463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:08.232659 13463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:08.235741 13463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:08.235948 13463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:08.236507 13463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:08.237287 13463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:08.237334 13463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:08.237383 13463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:08.237413 13463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:08.245492 13463 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:44359
I20260926 14:50:08.245527 13591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:44359 every 8 connection(s)
I20260926 14:50:08.246331 13463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:08.248134 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 13463
I20260926 14:50:08.248270 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:08.248487 12563 external_mini_cluster.cc:1428] Setting key 2d1e05fb2d7f304b18d06404bfe2352c
Sep 26 14:50:08 dist-test-slave-24mn krb5kdc[13088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434208, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:08.259138 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.248171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:34383 (local address 127.12.68.254:42115)
0926 14:50:08.248327 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:08.248332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:08.249217 (+ 885us) server_negotiation.cc:407] Connection header received
0926 14:50:08.250677 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:08.250680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:08.250733 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:08.250803 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:08.251934 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.252461 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.253555 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.253774 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.253855 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.256202 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:08.256205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:08.256207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:08.256230 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:08.257495 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:08.257934 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:08.257937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:08.257938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:08.258004 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:08.258267 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:08.258269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:08.258270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:08.258435 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:08.258506 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:08.258932 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 14:50:08.259020 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 14:50:08.259996 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.248568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42115 (local address 127.12.68.195:34383)
0926 14:50:08.249049 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:08.249105 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:08.250477 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:08.250982 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:08.250989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:08.251330 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:08.251783 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:08.251795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.252569 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.252572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:08.253442 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:08.253451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.253899 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.253903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:08.254077 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.254629 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:08.254644 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:08.256029 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:08.257600 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:08.257604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:08.257615 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:08.257862 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:08.258067 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:08.258070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:08.258071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:08.258203 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:08.258512 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:08.258517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:08.258754 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 14:50:08.258904 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 14:50:08.259134 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":137,"threads_started":1}
I20260926 14:50:08.261197 13592 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42115
I20260926 14:50:08.261432 13592 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:08.261935 13592 heartbeater.cc:511] Master 127.12.68.254:42115 requested a full tablet report, sending...
I20260926 14:50:08.262888 13133 ts_manager.cc:194] Registered new tserver with Master: 185827d21319457495ba3bc80fdc181d (127.12.68.195:44359)
I20260926 14:50:08.263417 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:34383
I20260926 14:50:08.273723 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:50:08.283715 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.277113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.12.68.254:42115)
0926 14:50:08.277248 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:08.277253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:08.277379 (+ 126us) server_negotiation.cc:407] Connection header received
0926 14:50:08.277495 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:08.277499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:08.277539 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:08.277599 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:08.278253 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.278756 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.279531 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.279734 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.279815 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.281051 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:08.281054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:08.281056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:08.281078 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:08.282258 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:08.282626 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:08.282629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:08.282631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:08.282702 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:08.282937 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:08.282941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:08.282942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:08.283141 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:08.283254 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:08.283469 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 14:50:08.283573 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:50:08.287968 13133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:08.290752 13133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:08.303714 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.298297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:44359 (local address 127.0.0.1:52524)
0926 14:50:08.299113 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:08.299160 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:08.299283 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:08.300003 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:08.300005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:08.300017 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:08.300160 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:08.300166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.301118 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.301122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:08.302260 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:08.302270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.303216 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.303219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:08.303322 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.303369 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 14:50:08.303463 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 14:50:08.303578 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":43,"thread_start_us":64,"threads_started":1}
I20260926 14:50:08.303723 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.298438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.12.68.195:44359)
0926 14:50:08.298876 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:08.298888 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:08.299425 (+ 537us) server_negotiation.cc:407] Connection header received
0926 14:50:08.299504 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:08.299510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:08.299703 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:08.299843 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:08.300265 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.301003 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.302389 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.303112 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.303201 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.303340 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:08.303498 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 14:50:08.303587 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":144,"threads_started":1}
I20260926 14:50:08.304003 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.298987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:34753 (local address 127.0.0.1:56090)
0926 14:50:08.299222 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:08.299233 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:08.299288 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:08.299974 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:08.299979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:08.300001 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:08.300154 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:08.300161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.301685 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.301688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:08.302588 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:08.302597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.303634 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.303638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:08.303719 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.303743 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:50:08.303785 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 14:50:08.303851 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":6,"thread_start_us":68,"threads_started":1}
I20260926 14:50:08.304457 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.298712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.12.68.194:43149)
0926 14:50:08.299071 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:08.299087 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:08.299311 (+ 224us) server_negotiation.cc:407] Connection header received
0926 14:50:08.299524 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:08.299530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:08.299721 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:08.299857 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:08.300322 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.301477 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.302891 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.303669 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.303755 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.303925 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:08.304230 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 14:50:08.304323 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
I20260926 14:50:08.304535 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.299151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56090 (local address 127.12.68.193:34753)
0926 14:50:08.299552 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:08.299563 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:08.299582 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:50:08.299641 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:08.299645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:08.299782 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:08.300305 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:08.300474 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.301562 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.302710 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.303498 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.304155 (+ 657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.304286 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:08.304327 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 14:50:08.304415 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":115,"threads_started":1}
I20260926 14:50:08.304785 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.298425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:43149 (local address 127.0.0.1:56694)
0926 14:50:08.299211 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:08.299224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:08.299384 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:08.300076 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:08.300078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:08.300094 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:08.300204 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:08.300210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.301611 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.301614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:08.302734 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:08.302743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.303993 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.303997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:08.304089 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.304110 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:50:08.304632 (+ 522us) client_negotiation.cc:240] Negotiation successful
0926 14:50:08.304699 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":6,"thread_start_us":97,"threads_started":1}
I20260926 14:50:08.306751 13526 tablet_service.cc:1467] Processing CreateTablet for tablet 945db554e78b46c5a4ce79faf02c0cfa (DEFAULT_TABLE table=test-table [id=b51d5be0367745599eb233ef8d16adcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:08.307117 13392 tablet_service.cc:1467] Processing CreateTablet for tablet 945db554e78b46c5a4ce79faf02c0cfa (DEFAULT_TABLE table=test-table [id=b51d5be0367745599eb233ef8d16adcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:08.307212 13258 tablet_service.cc:1467] Processing CreateTablet for tablet 945db554e78b46c5a4ce79faf02c0cfa (DEFAULT_TABLE table=test-table [id=b51d5be0367745599eb233ef8d16adcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:08.307852 13526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945db554e78b46c5a4ce79faf02c0cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:08.308003 13392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945db554e78b46c5a4ce79faf02c0cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:08.308168 13258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945db554e78b46c5a4ce79faf02c0cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:08.313486 13609 tablet_bootstrap.cc:492] T 945db554e78b46c5a4ce79faf02c0cfa P 7436ab04f7af4cde8cefd904be89cc5c: Bootstrap starting.
I20260926 14:50:08.314895 13610 tablet_bootstrap.cc:492] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b: Bootstrap starting.
I20260926 14:50:08.314895 13611 tablet_bootstrap.cc:492] T 945db554e78b46c5a4ce79faf02c0cfa P 185827d21319457495ba3bc80fdc181d: Bootstrap starting.
I20260926 14:50:08.316645 13609 tablet_bootstrap.cc:654] T 945db554e78b46c5a4ce79faf02c0cfa P 7436ab04f7af4cde8cefd904be89cc5c: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:08.316911 13611 tablet_bootstrap.cc:654] T 945db554e78b46c5a4ce79faf02c0cfa P 185827d21319457495ba3bc80fdc181d: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:08.317273 13610 tablet_bootstrap.cc:654] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:08.317349 13609 log.cc:824] T 945db554e78b46c5a4ce79faf02c0cfa P 7436ab04f7af4cde8cefd904be89cc5c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:08.318038 13610 log.cc:824] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:08.318038 13611 log.cc:824] T 945db554e78b46c5a4ce79faf02c0cfa P 185827d21319457495ba3bc80fdc181d: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:08.319182 13609 tablet_bootstrap.cc:492] T 945db554e78b46c5a4ce79faf02c0cfa P 7436ab04f7af4cde8cefd904be89cc5c: No bootstrap required, opened a new log
I20260926 14:50:08.319382 13609 ts_tablet_manager.cc:1397] T 945db554e78b46c5a4ce79faf02c0cfa P 7436ab04f7af4cde8cefd904be89cc5c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:50:08.319799 13611 tablet_bootstrap.cc:492] T 945db554e78b46c5a4ce79faf02c0cfa P 185827d21319457495ba3bc80fdc181d: No bootstrap required, opened a new log
I20260926 14:50:08.319998 13611 ts_tablet_manager.cc:1397] T 945db554e78b46c5a4ce79faf02c0cfa P 185827d21319457495ba3bc80fdc181d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:50:08.320394 13610 tablet_bootstrap.cc:492] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b: No bootstrap required, opened a new log
I20260926 14:50:08.320549 13610 ts_tablet_manager.cc:1397] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:50:08.322634 13611 raft_consensus.cc:348] T 945db554e78b46c5a4ce79faf02c0cfa P 185827d21319457495ba3bc80fdc181d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "185827d21319457495ba3bc80fdc181d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44359 } } peers { permanent_uuid: "7436ab04f7af4cde8cefd904be89cc5c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43149 } } peers { permanent_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34753 } }
I20260926 14:50:08.322849 13611 raft_consensus.cc:374] T 945db554e78b46c5a4ce79faf02c0cfa P 185827d21319457495ba3bc80fdc181d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:08.323053 13611 raft_consensus.cc:729] T 945db554e78b46c5a4ce79faf02c0cfa P 185827d21319457495ba3bc80fdc181d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185827d21319457495ba3bc80fdc181d, State: Initialized, Role: FOLLOWER
I20260926 14:50:08.323122 13609 raft_consensus.cc:348] T 945db554e78b46c5a4ce79faf02c0cfa P 7436ab04f7af4cde8cefd904be89cc5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "185827d21319457495ba3bc80fdc181d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44359 } } peers { permanent_uuid: "7436ab04f7af4cde8cefd904be89cc5c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43149 } } peers { permanent_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34753 } }
I20260926 14:50:08.323415 13609 raft_consensus.cc:374] T 945db554e78b46c5a4ce79faf02c0cfa P 7436ab04f7af4cde8cefd904be89cc5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:08.323468 13609 raft_consensus.cc:729] T 945db554e78b46c5a4ce79faf02c0cfa P 7436ab04f7af4cde8cefd904be89cc5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7436ab04f7af4cde8cefd904be89cc5c, State: Initialized, Role: FOLLOWER
I20260926 14:50:08.323510 13611 consensus_queue.cc:260] T 945db554e78b46c5a4ce79faf02c0cfa P 185827d21319457495ba3bc80fdc181d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "185827d21319457495ba3bc80fdc181d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44359 } } peers { permanent_uuid: "7436ab04f7af4cde8cefd904be89cc5c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43149 } } peers { permanent_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34753 } }
I20260926 14:50:08.323925 13609 consensus_queue.cc:260] T 945db554e78b46c5a4ce79faf02c0cfa P 7436ab04f7af4cde8cefd904be89cc5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "185827d21319457495ba3bc80fdc181d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44359 } } peers { permanent_uuid: "7436ab04f7af4cde8cefd904be89cc5c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43149 } } peers { permanent_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34753 } }
I20260926 14:50:08.324123 13592 heartbeater.cc:503] Master 127.12.68.254:42115 was elected leader, sending a full tablet report...
I20260926 14:50:08.324462 13611 ts_tablet_manager.cc:1428] T 945db554e78b46c5a4ce79faf02c0cfa P 185827d21319457495ba3bc80fdc181d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 14:50:08.324815 13609 ts_tablet_manager.cc:1428] T 945db554e78b46c5a4ce79faf02c0cfa P 7436ab04f7af4cde8cefd904be89cc5c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:50:08.324662 13610 raft_consensus.cc:348] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "185827d21319457495ba3bc80fdc181d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44359 } } peers { permanent_uuid: "7436ab04f7af4cde8cefd904be89cc5c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43149 } } peers { permanent_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34753 } }
I20260926 14:50:08.324910 13610 raft_consensus.cc:374] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:08.324952 13610 raft_consensus.cc:729] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0642ecccb5489d8d3cd19b4685bf5b, State: Initialized, Role: FOLLOWER
I20260926 14:50:08.324970 13458 heartbeater.cc:503] Master 127.12.68.254:42115 was elected leader, sending a full tablet report...
I20260926 14:50:08.325309 13610 consensus_queue.cc:260] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "185827d21319457495ba3bc80fdc181d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44359 } } peers { permanent_uuid: "7436ab04f7af4cde8cefd904be89cc5c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43149 } } peers { permanent_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34753 } }
I20260926 14:50:08.325878 13324 heartbeater.cc:503] Master 127.12.68.254:42115 was elected leader, sending a full tablet report...
I20260926 14:50:08.326156 13610 ts_tablet_manager.cc:1428] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:50:08.332563 13617 raft_consensus.cc:482] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:08.332696 13617 raft_consensus.cc:504] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "185827d21319457495ba3bc80fdc181d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44359 } } peers { permanent_uuid: "7436ab04f7af4cde8cefd904be89cc5c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43149 } } peers { permanent_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34753 } }
I20260926 14:50:08.333626 13617 leader_election.cc:288] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 185827d21319457495ba3bc80fdc181d (127.12.68.195:44359), 7436ab04f7af4cde8cefd904be89cc5c (127.12.68.194:43149)
I20260926 14:50:08.337327 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.333935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:42563 (local address 127.12.68.195:44359)
0926 14:50:08.334034 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:08.334039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:08.334050 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:50:08.334088 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:08.334091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:08.334123 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:08.334179 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:08.334443 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.335020 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.336248 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.336806 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.336941 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.337001 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:08.337161 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 14:50:08.337218 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:50:08.337339 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.333808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:44359 (local address 127.12.68.193:42563)
0926 14:50:08.333906 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:08.333921 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:08.333994 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:08.334196 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:08.334198 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:08.334213 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:08.334315 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:08.334321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.335112 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.335115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:08.336153 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:08.336163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.336937 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.336940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:08.337035 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.337062 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 14:50:08.337146 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 14:50:08.337211 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 14:50:08.337486 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.333896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:37225 (local address 127.12.68.194:43149)
0926 14:50:08.334002 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:08.334006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:08.334273 (+ 267us) server_negotiation.cc:407] Connection header received
0926 14:50:08.334362 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:08.334365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:08.334407 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:08.334469 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:08.334691 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.335208 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.336420 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.336984 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.337086 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.337120 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:08.337307 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 14:50:08.337383 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:50:08.337495 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.333821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:43149 (local address 127.12.68.193:37225)
0926 14:50:08.334205 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:08.334215 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:08.334298 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:08.334483 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:08.334485 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:08.334500 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:08.334603 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:08.334609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.335311 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.335314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:08.336317 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:08.336326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.337092 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.337095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:08.337190 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.337209 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 14:50:08.337281 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 14:50:08.337331 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":99,"thread_start_us":113,"threads_started":1}
I20260926 14:50:08.337976 13412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945db554e78b46c5a4ce79faf02c0cfa" candidate_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7436ab04f7af4cde8cefd904be89cc5c" is_pre_election: true
I20260926 14:50:08.337961 13546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945db554e78b46c5a4ce79faf02c0cfa" candidate_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "185827d21319457495ba3bc80fdc181d" is_pre_election: true
I20260926 14:50:08.338219 13546 raft_consensus.cc:2454] T 945db554e78b46c5a4ce79faf02c0cfa P 185827d21319457495ba3bc80fdc181d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0642ecccb5489d8d3cd19b4685bf5b in term 0.
I20260926 14:50:08.338219 13412 raft_consensus.cc:2454] T 945db554e78b46c5a4ce79faf02c0cfa P 7436ab04f7af4cde8cefd904be89cc5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0642ecccb5489d8d3cd19b4685bf5b in term 0.
I20260926 14:50:08.338672 13214 leader_election.cc:302] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 185827d21319457495ba3bc80fdc181d, 7436ab04f7af4cde8cefd904be89cc5c, ee0642ecccb5489d8d3cd19b4685bf5b; no voters:
I20260926 14:50:08.338892 13617 raft_consensus.cc:2781] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:08.338966 13617 raft_consensus.cc:482] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:08.339020 13617 raft_consensus.cc:3037] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:08.339972 13617 raft_consensus.cc:504] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "185827d21319457495ba3bc80fdc181d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44359 } } peers { permanent_uuid: "7436ab04f7af4cde8cefd904be89cc5c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43149 } } peers { permanent_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34753 } }
I20260926 14:50:08.340289 13617 leader_election.cc:288] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [CANDIDATE]: Term 1 election: Requested vote from peers 185827d21319457495ba3bc80fdc181d (127.12.68.195:44359), 7436ab04f7af4cde8cefd904be89cc5c (127.12.68.194:43149)
I20260926 14:50:08.340564 13546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945db554e78b46c5a4ce79faf02c0cfa" candidate_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "185827d21319457495ba3bc80fdc181d"
I20260926 14:50:08.340590 13412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945db554e78b46c5a4ce79faf02c0cfa" candidate_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7436ab04f7af4cde8cefd904be89cc5c"
I20260926 14:50:08.340675 13546 raft_consensus.cc:3037] T 945db554e78b46c5a4ce79faf02c0cfa P 185827d21319457495ba3bc80fdc181d [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:08.340675 13412 raft_consensus.cc:3037] T 945db554e78b46c5a4ce79faf02c0cfa P 7436ab04f7af4cde8cefd904be89cc5c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:08.341850 13412 raft_consensus.cc:2454] T 945db554e78b46c5a4ce79faf02c0cfa P 7436ab04f7af4cde8cefd904be89cc5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0642ecccb5489d8d3cd19b4685bf5b in term 1.
I20260926 14:50:08.341850 13546 raft_consensus.cc:2454] T 945db554e78b46c5a4ce79faf02c0cfa P 185827d21319457495ba3bc80fdc181d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0642ecccb5489d8d3cd19b4685bf5b in term 1.
I20260926 14:50:08.342125 13212 leader_election.cc:302] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 185827d21319457495ba3bc80fdc181d, 7436ab04f7af4cde8cefd904be89cc5c, ee0642ecccb5489d8d3cd19b4685bf5b; no voters:
I20260926 14:50:08.342278 13617 raft_consensus.cc:2781] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:08.342474 13617 raft_consensus.cc:686] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [term 1 LEADER]: Becoming Leader. State: Replica: ee0642ecccb5489d8d3cd19b4685bf5b, State: Running, Role: LEADER
I20260926 14:50:08.342742 13617 consensus_queue.cc:237] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "185827d21319457495ba3bc80fdc181d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44359 } } peers { permanent_uuid: "7436ab04f7af4cde8cefd904be89cc5c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43149 } } peers { permanent_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34753 } }
I20260926 14:50:08.345527 13130 catalog_manager.cc:5495] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0642ecccb5489d8d3cd19b4685bf5b (127.12.68.193). New cstate: current_term: 1 leader_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "185827d21319457495ba3bc80fdc181d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7436ab04f7af4cde8cefd904be89cc5c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0642ecccb5489d8d3cd19b4685bf5b" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34753 } health_report { overall_health: HEALTHY } } }
W20260926 14:50:08.352903 13325 tablet.cc:2369] T 945db554e78b46c5a4ce79faf02c0cfa P ee0642ecccb5489d8d3cd19b4685bf5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 14:50:08 dist-test-slave-24mn krb5kdc[13088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.193@KRBTEST.COM
I20260926 14:50:08.368158 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.359964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.12.68.193:34753)
0926 14:50:08.360069 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:08.360073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:08.360221 (+ 148us) server_negotiation.cc:407] Connection header received
0926 14:50:08.360315 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:08.360317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:08.360354 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:08.360419 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:08.361128 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.361629 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.362425 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.362638 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.362712 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.364919 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:08.364928 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:08.364940 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:08.364963 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:08.366480 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:08.366957 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:08.366963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:08.366964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:08.367064 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:08.367389 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:08.367392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:08.367393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:08.367624 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:08.367777 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:08.367947 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 14:50:08.368041 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 14:50:08 dist-test-slave-24mn krb5kdc[13088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434208, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:50:08 dist-test-slave-24mn krb5kdc[13088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434208, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.68.193@KRBTEST.COM
I20260926 14:50:08.392670 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.385298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56116 (local address 127.12.68.193:34753)
0926 14:50:08.385452 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:08.385456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:08.385524 (+ 68us) server_negotiation.cc:407] Connection header received
0926 14:50:08.385632 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:08.385634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:08.385676 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:08.385742 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:08.386332 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.386864 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.387632 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.387841 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.387960 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.390105 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:08.390108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:08.390110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:08.390132 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:08.391321 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:08.391640 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:08.391643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:08.391644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:08.391709 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:08.391978 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:08.391980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:08.391981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:08.392123 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:08.392219 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:08.392430 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 14:50:08.392517 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 14:50:08 dist-test-slave-24mn krb5kdc[13088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:50:08 dist-test-slave-24mn krb5kdc[13088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.193@KRBTEST.COM
I20260926 14:50:08.415084 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:08.407963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56124 (local address 127.12.68.193:34753)
0926 14:50:08.408067 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:08.408070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:08.408199 (+ 129us) server_negotiation.cc:407] Connection header received
0926 14:50:08.408298 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:08.408300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:08.408347 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:08.408411 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:08.408967 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.409457 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.410184 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:08.410389 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:08.410452 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:08.412489 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:08.412492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:08.412493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:08.412514 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:08.413685 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:08.414066 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:08.414068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:08.414069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:08.414136 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:08.414314 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:08.414316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:08.414317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:08.414461 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:08.414558 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:08.414837 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 14:50:08.414933 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 14:50:08.416440 13238 tablet_service.cc:2908] Rejecting scan request for tablet 945db554e78b46c5a4ce79faf02c0cfa: Uninitialized: clean time has not yet been initialized
I20260926 14:50:08.418581 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 13195
I20260926 14:50:08.423590 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 13329
I20260926 14:50:08.427980 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 13463
I20260926 14:50:08.432139 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 13101
2026-09-26T14:50:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2055 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:08 dist-test-slave-24mn krb5kdc[13633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:08 dist-test-slave-24mn krb5kdc[13633](info): set up 2 sockets
Sep 26 14:50:08 dist-test-slave-24mn krb5kdc[13633](info): commencing operation
krb5kdc: starting...
W20260926 14:50:09.371362 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:50:09 dist-test-slave-24mn krb5kdc[13633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:50:09.476578 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:41287
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:41287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:09.578872 13646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:09.579107 13646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:09.579169 13646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:09.583026 13646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:09.583076 13646 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 14:50:09.583098 13646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:09.583120 13646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:09.583158 13646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:09.583186 13646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:09.588495 13646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:41287
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:41287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:09.589419 13646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:09.590328 13646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:09.598095 13651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:09.598090 13654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:09.598134 13646 server_base.cc:1034] running on GCE node
W20260926 14:50:09.598090 13652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:09.598743 13646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:09.599701 13646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:09.600849 13646 hybrid_clock.cc:648] HybridClock initialized: now 1790434209600827 us; error 34 us; skew 500 ppm
Sep 26 14:50:09 dist-test-slave-24mn krb5kdc[13633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:09.608479 13646 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:50:09.609292 13646 webserver.cc:460] Webserver started at http://127.12.68.254:39045/ using document root <none> and password file <none>
I20260926 14:50:09.609789 13646 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:09.609851 13646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:09.610021 13646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:09.611955 13646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "3c5b9f00e888450e84d24b825e888190"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-24mn"
server_key: "bb8d1931fa816d0f40a11b8252ede87a"
server_key_iv: "bec9248e765b5f06be7390a83a8a9d97"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.612578 13646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "3c5b9f00e888450e84d24b825e888190"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-24mn"
server_key: "bb8d1931fa816d0f40a11b8252ede87a"
server_key_iv: "bec9248e765b5f06be7390a83a8a9d97"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.615906 13646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260926 14:50:09.617995 13661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.619067 13646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:09.619172 13646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "3c5b9f00e888450e84d24b825e888190"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-24mn"
server_key: "bb8d1931fa816d0f40a11b8252ede87a"
server_key_iv: "bec9248e765b5f06be7390a83a8a9d97"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.619256 13646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:09.639171 13646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:09.641975 13646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:09.642140 13646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:09.649325 13646 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:41287
I20260926 14:50:09.649333 13723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:41287 every 8 connection(s)
I20260926 14:50:09.650209 13646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:09.651167 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 13646
I20260926 14:50:09.651280 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:09.651512 12563 external_mini_cluster.cc:1428] Setting key 91a7331bd0ab47256a8b31a878c7c250
I20260926 14:50:09.653205 13724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:09.659246 13724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190: Bootstrap starting.
Sep 26 14:50:09 dist-test-slave-24mn krb5kdc[13633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:09.661440 13724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:09.662412 13724 log.cc:824] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:09.664178 13724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190: No bootstrap required, opened a new log
I20260926 14:50:09.664724 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.652759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.12.68.254:41287)
0926 14:50:09.653137 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.653162 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.653196 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:50:09.653831 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.653848 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.654187 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.654471 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:09.655303 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.656363 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.657471 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.657919 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.658029 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.660566 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:09.660571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:09.660583 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:09.660609 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:09.662102 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.662505 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.662510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.662516 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.662634 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.662880 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.662884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.662885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.663262 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:09.663428 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.663699 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.664002 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":108,"threads_started":1}
I20260926 14:50:09.667273 13724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5b9f00e888450e84d24b825e888190" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41287 } }
I20260926 14:50:09.667474 13724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:09.667536 13724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5b9f00e888450e84d24b825e888190, State: Initialized, Role: FOLLOWER
I20260926 14:50:09.667938 13724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5b9f00e888450e84d24b825e888190" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41287 } }
I20260926 14:50:09.668063 13724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:09.668124 13724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:09.668216 13724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:09.669102 13724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5b9f00e888450e84d24b825e888190" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41287 } }
I20260926 14:50:09.669431 13724 leader_election.cc:302] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c5b9f00e888450e84d24b825e888190; no voters:
I20260926 14:50:09.669716 13724 leader_election.cc:288] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:09.669811 13729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:09.670012 13729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5b9f00e888450e84d24b825e888190, State: Running, Role: LEADER
I20260926 14:50:09.670342 13729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5b9f00e888450e84d24b825e888190" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41287 } }
I20260926 14:50:09.670773 13724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:09.671729 13731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c5b9f00e888450e84d24b825e888190. Latest consensus state: current_term: 1 leader_uuid: "3c5b9f00e888450e84d24b825e888190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5b9f00e888450e84d24b825e888190" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41287 } } }
I20260926 14:50:09.671844 13731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:09.672170 13730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c5b9f00e888450e84d24b825e888190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5b9f00e888450e84d24b825e888190" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41287 } } }
I20260926 14:50:09.672309 13730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:09.672206 13738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:09.675045 13738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:09.680050 13738 catalog_manager.cc:1347] Generated new cluster ID: 986c3c3799fd4386bd7387ae0b589e30
I20260926 14:50:09.680104 13738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:09.696842 13738 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:09.697717 13738 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:09.704650 13738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190: Generated new TSK 0
I20260926 14:50:09.705262 13738 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:50:09.728591 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:41287
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:09.832405 13750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:09.832636 13750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:09.832675 13750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:09.836553 13750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:09.836611 13750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:09.836647 13750 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:09.836675 13750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:09.841612 13750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.68.254:41287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:09.842509 13750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:09.843463 13750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:09.852283 13756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:09.852349 13750 server_base.cc:1034] running on GCE node
W20260926 14:50:09.852289 13755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:09.852304 13758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:09.852892 13750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:09.853487 13750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:09.854630 13750 hybrid_clock.cc:648] HybridClock initialized: now 1790434209854609 us; error 30 us; skew 500 ppm
Sep 26 14:50:09 dist-test-slave-24mn krb5kdc[13633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:09.857096 13750 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:50:09.857833 13750 webserver.cc:460] Webserver started at http://127.12.68.193:38585/ using document root <none> and password file <none>
I20260926 14:50:09.858322 13750 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:09.858382 13750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:09.858548 13750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:09.860455 13750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "7a58b3da49ec4eafb0630b40bea6e7e0"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-24mn"
server_key: "6e590b84dddd81ca12cd9b3fb4fbed10"
server_key_iv: "681be2e906bb2212b57f9bb5ba5f06b1"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.860944 13750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "7a58b3da49ec4eafb0630b40bea6e7e0"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-24mn"
server_key: "6e590b84dddd81ca12cd9b3fb4fbed10"
server_key_iv: "681be2e906bb2212b57f9bb5ba5f06b1"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.864252 13750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:09.866362 13765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.867431 13750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:09.867527 13750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "7a58b3da49ec4eafb0630b40bea6e7e0"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-24mn"
server_key: "6e590b84dddd81ca12cd9b3fb4fbed10"
server_key_iv: "681be2e906bb2212b57f9bb5ba5f06b1"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.867607 13750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:09.889318 13750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:09.891820 13750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:09.891988 13750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:09.893214 13750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:09.893275 13750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.893373 13750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:09.893400 13750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:50:09 dist-test-slave-24mn krb5kdc[13633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:09.902822 13750 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:33419
I20260926 14:50:09.902908 13880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:33419 every 8 connection(s)
I20260926 14:50:09.903729 13750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:09.905239 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.893969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:53995 (local address 127.12.68.254:41287)
0926 14:50:09.894095 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.894100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.894927 (+ 827us) server_negotiation.cc:407] Connection header received
0926 14:50:09.895976 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.895980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.896037 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.896129 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:09.897295 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.897921 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.899004 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.899286 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.899371 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.901733 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:09.901737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:09.901740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:09.901769 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:09.903249 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.903734 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.903737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.903739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.903837 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.904214 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.904217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.904219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.904431 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:09.904525 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.904956 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.905086 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:50:09.905874 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.894326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41287 (local address 127.12.68.193:53995)
0926 14:50:09.894775 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.894821 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.895742 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.896324 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.896332 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.896697 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:09.897158 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.897171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.898038 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.898041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.898883 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.898892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.899434 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.899438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.899590 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.900111 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:09.900128 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:09.901592 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:09.903374 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.903380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.903393 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.903644 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.903956 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.903958 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.903960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.904103 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.904519 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:09.904525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:09.904742 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.904895 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.905170 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
I20260926 14:50:09.906970 13881 heartbeater.cc:347] Connected to a master server at 127.12.68.254:41287
I20260926 14:50:09.907095 13881 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:09.907537 13881 heartbeater.cc:511] Master 127.12.68.254:41287 requested a full tablet report, sending...
I20260926 14:50:09.908984 13678 ts_manager.cc:194] Registered new tserver with Master: 7a58b3da49ec4eafb0630b40bea6e7e0 (127.12.68.193:33419)
I20260926 14:50:09.910014 13678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:53995
I20260926 14:50:09.913923 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 13750
I20260926 14:50:09.914026 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:09.914263 12563 external_mini_cluster.cc:1428] Setting key 447321aef7f7abe038e7b1159ed1c73a
I20260926 14:50:09.914974 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.908752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41287 (local address 127.12.68.193:49269)
0926 14:50:09.909048 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.909061 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.909159 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.909353 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.909356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.909516 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:09.909596 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.909601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.910311 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.910314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.910965 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.910974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.911370 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.911374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.911496 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.911896 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:09.911910 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:09.912191 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:09.913597 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.913600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.913602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.913828 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.914127 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.914130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.914131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.914205 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.914544 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:09.914547 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:09.914624 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.914716 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.914839 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":4,"thread_start_us":85,"threads_started":1}
I20260926 14:50:09.915000 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.908824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:49269 (local address 127.12.68.254:41287)
0926 14:50:09.908932 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.908936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.909127 (+ 191us) server_negotiation.cc:407] Connection header received
0926 14:50:09.909223 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.909226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.909264 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.909331 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:09.909691 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.910200 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.911064 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.911273 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.911359 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.912314 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:09.912317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:09.912320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:09.912341 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:09.913504 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.913927 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.913930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.913931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.914022 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.914285 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.914289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.914290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.914461 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:09.914541 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.914767 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.914874 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:50:09.938207 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:41287
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:10.038836 13890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:10.039057 13890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:10.039110 13890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:10.042956 13890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:10.043004 13890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:10.043058 13890 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:10.043110 13890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:10.048013 13890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.68.254:41287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:10.048913 13890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:10.049808 13890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:10.058295 13896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:10.058323 13895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:10.058342 13890 server_base.cc:1034] running on GCE node
W20260926 14:50:10.058348 13898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:10.058859 13890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:10.059404 13890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:10.060549 13890 hybrid_clock.cc:648] HybridClock initialized: now 1790434210060531 us; error 33 us; skew 500 ppm
Sep 26 14:50:10 dist-test-slave-24mn krb5kdc[13633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434210, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:10.062932 13890 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:50:10.063673 13890 webserver.cc:460] Webserver started at http://127.12.68.194:46647/ using document root <none> and password file <none>
I20260926 14:50:10.064190 13890 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:10.064245 13890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:10.064420 13890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:10.066243 13890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "b98c3dce44e246f1bc6e32e4328a5cdd"
format_stamp: "Formatted at 2026-09-26 14:50:10 on dist-test-slave-24mn"
server_key: "7e6cf798b341d8ef0864dda7db664c7c"
server_key_iv: "14275c703493c7cee5bdcd694c829768"
server_key_version: "encryptionkey@0"
I20260926 14:50:10.066744 13890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "b98c3dce44e246f1bc6e32e4328a5cdd"
format_stamp: "Formatted at 2026-09-26 14:50:10 on dist-test-slave-24mn"
server_key: "7e6cf798b341d8ef0864dda7db664c7c"
server_key_iv: "14275c703493c7cee5bdcd694c829768"
server_key_version: "encryptionkey@0"
I20260926 14:50:10.070112 13890 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:50:10.072283 13905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:10.073388 13890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:10.073480 13890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "b98c3dce44e246f1bc6e32e4328a5cdd"
format_stamp: "Formatted at 2026-09-26 14:50:10 on dist-test-slave-24mn"
server_key: "7e6cf798b341d8ef0864dda7db664c7c"
server_key_iv: "14275c703493c7cee5bdcd694c829768"
server_key_version: "encryptionkey@0"
I20260926 14:50:10.073553 13890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:10.095680 13890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:10.098708 13890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:10.098845 13890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:10.100092 13890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:10.100140 13890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:10.100176 13890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:10.100212 13890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:50:10 dist-test-slave-24mn krb5kdc[13633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434210, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:10.109334 13890 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:38989
I20260926 14:50:10.109428 14020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:38989 every 8 connection(s)
I20260926 14:50:10.110154 13890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:10.112898 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.100811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:40561 (local address 127.12.68.254:41287)
0926 14:50:10.100921 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.100924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.101879 (+ 955us) server_negotiation.cc:407] Connection header received
0926 14:50:10.102893 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.102896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.102954 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.103027 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:10.104454 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.105253 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.106669 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.106968 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.107056 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.109747 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:10.109750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:10.109752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:10.109778 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:10.111224 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.111628 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.111630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.111631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.111715 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.112080 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.112082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.112083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.112245 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:10.112316 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.112693 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.112786 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:50:10.112993 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 13890
I20260926 14:50:10.113098 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:10.113353 12563 external_mini_cluster.cc:1428] Setting key 5446ddb2996bf2c5224ef78df14c6656
I20260926 14:50:10.113765 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.101302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41287 (local address 127.12.68.194:40561)
0926 14:50:10.101747 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.101785 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.102603 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.103290 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.103298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.103698 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:10.104295 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.104314 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.105367 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.105370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.106557 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.106569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.107105 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.107109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.107289 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.107818 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:10.107836 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:10.109594 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:10.111313 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:10.111317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:10.111325 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:10.111553 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:10.111830 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:10.111832 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:10.111834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:10.112009 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:10.112315 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:10.112319 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:10.112510 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.112650 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.112889 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":130,"threads_started":1}
I20260926 14:50:10.114953 14021 heartbeater.cc:347] Connected to a master server at 127.12.68.254:41287
I20260926 14:50:10.115079 14021 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:10.115545 14021 heartbeater.cc:511] Master 127.12.68.254:41287 requested a full tablet report, sending...
I20260926 14:50:10.116555 13678 ts_manager.cc:194] Registered new tserver with Master: b98c3dce44e246f1bc6e32e4328a5cdd (127.12.68.194:38989)
I20260926 14:50:10.117255 13678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:40561
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
I20260926 14:50:10.123158 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.116964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41287 (local address 127.12.68.194:60297)
0926 14:50:10.117190 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.117201 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.117281 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.117537 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.117540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.117740 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:10.117851 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.117857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.118624 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.118626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.119202 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.119212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.119693 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.119697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.119781 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.120177 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:10.120187 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:10.120449 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:10.121830 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:10.121833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:10.121835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:10.122014 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:10.122257 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:10.122260 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:10.122262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:10.122335 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:10.122711 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:10.122714 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:10.122794 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.122876 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.122993 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":83,"threads_started":1}
I20260926 14:50:10.123217 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.116905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:60297 (local address 127.12.68.254:41287)
0926 14:50:10.116990 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.116994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.117263 (+ 269us) server_negotiation.cc:407] Connection header received
0926 14:50:10.117401 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.117404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.117442 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.117511 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:10.117961 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.118511 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.119350 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.119601 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.119684 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.120565 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:10.120569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:10.120571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:10.120594 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:10.121738 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.122096 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.122099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.122101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.122180 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.122410 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.122413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.122415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.122634 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:10.122719 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.122936 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.123026 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:50:10.136554 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:41287
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:10.237043 14030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:10.237259 14030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:10.237298 14030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:10.241073 14030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:10.241113 14030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:10.241148 14030 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:10.241178 14030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:10.245941 14030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.68.254:41287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:10.246730 14030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:10.247552 14030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:10.255993 14038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:10.255982 14035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:10.256028 14036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:10.256273 14030 server_base.cc:1034] running on GCE node
I20260926 14:50:10.256582 14030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:10.257108 14030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:10.258288 14030 hybrid_clock.cc:648] HybridClock initialized: now 1790434210258274 us; error 24 us; skew 500 ppm
Sep 26 14:50:10 dist-test-slave-24mn krb5kdc[13633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434210, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:10.260602 14030 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:50:10.261337 14030 webserver.cc:460] Webserver started at http://127.12.68.195:35105/ using document root <none> and password file <none>
I20260926 14:50:10.261783 14030 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:10.261832 14030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:10.261979 14030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:10.263751 14030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "78497612d7c54fe49f891cf072745241"
format_stamp: "Formatted at 2026-09-26 14:50:10 on dist-test-slave-24mn"
server_key: "ece24365a68738f9fd5a4525dbc41a43"
server_key_iv: "51b985dcab69c86b59ad3761912ba809"
server_key_version: "encryptionkey@0"
I20260926 14:50:10.264252 14030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "78497612d7c54fe49f891cf072745241"
format_stamp: "Formatted at 2026-09-26 14:50:10 on dist-test-slave-24mn"
server_key: "ece24365a68738f9fd5a4525dbc41a43"
server_key_iv: "51b985dcab69c86b59ad3761912ba809"
server_key_version: "encryptionkey@0"
I20260926 14:50:10.267500 14030 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:10.269630 14045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:10.270823 14030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:10.270907 14030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "78497612d7c54fe49f891cf072745241"
format_stamp: "Formatted at 2026-09-26 14:50:10 on dist-test-slave-24mn"
server_key: "ece24365a68738f9fd5a4525dbc41a43"
server_key_iv: "51b985dcab69c86b59ad3761912ba809"
server_key_version: "encryptionkey@0"
I20260926 14:50:10.270979 14030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:10.282333 14030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:10.284759 14030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:10.284888 14030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:10.286098 14030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:10.286144 14030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:10.286192 14030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:10.286238 14030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:50:10 dist-test-slave-24mn krb5kdc[13633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434210, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:10.295650 14030 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:38303
I20260926 14:50:10.295840 14160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:38303 every 8 connection(s)
I20260926 14:50:10.296491 14030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:10.298234 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.286723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:49997 (local address 127.12.68.254:41287)
0926 14:50:10.286837 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.286842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.288302 (+ 1460us) server_negotiation.cc:407] Connection header received
0926 14:50:10.289345 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.289348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.289407 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.289475 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:10.290607 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.291176 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.292456 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.292676 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.292760 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.295052 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:10.295056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:10.295059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:10.295086 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:10.296442 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.296937 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.296940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.296942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.297028 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.297364 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.297367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.297368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.297536 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:10.297614 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.297996 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.298122 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:50:10.298935 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.287668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41287 (local address 127.12.68.195:49997)
0926 14:50:10.288174 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.288212 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.289152 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.289641 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.289647 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.290036 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:10.290467 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.290480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.291286 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.291289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.292345 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.292355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.292812 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.292816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.292973 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.293420 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:10.293435 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:10.294904 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:10.296542 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:10.296548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:10.296561 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:10.296821 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:10.297119 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:10.297122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:10.297124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:10.297279 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:10.297617 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:10.297621 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:10.297817 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.297956 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.298163 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":144,"threads_started":1}
I20260926 14:50:10.300041 14161 heartbeater.cc:347] Connected to a master server at 127.12.68.254:41287
I20260926 14:50:10.300158 14161 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:10.300573 14161 heartbeater.cc:511] Master 127.12.68.254:41287 requested a full tablet report, sending...
I20260926 14:50:10.301149 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 14030
I20260926 14:50:10.301256 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:10.301437 13677 ts_manager.cc:194] Registered new tserver with Master: 78497612d7c54fe49f891cf072745241 (127.12.68.195:38303)
I20260926 14:50:10.301523 12563 external_mini_cluster.cc:1428] Setting key c6c8694f8cad12d3d7706f0ff1ee3069
I20260926 14:50:10.302119 13677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:49997
I20260926 14:50:10.303354 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:50:10.308487 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.302068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:55109 (local address 127.12.68.254:41287)
0926 14:50:10.302165 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.302169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.302385 (+ 216us) server_negotiation.cc:407] Connection header received
0926 14:50:10.302543 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.302546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.302589 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.302656 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:10.303037 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.303641 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.304442 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.304652 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.304729 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.305632 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:10.305636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:10.305638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:10.305661 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:10.307010 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.307456 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.307459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.307461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.307542 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.307766 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.307769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.307771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.307980 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:10.308073 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.308309 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.308389 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 14:50:10.308655 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.301915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41287 (local address 127.12.68.195:55109)
0926 14:50:10.302315 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.302327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.302445 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.302669 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.302672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.302840 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:10.302944 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.302949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.303757 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.303761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.304345 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.304352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.304748 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.304752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.304838 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.305211 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:10.305225 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:10.305519 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:10.307122 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:10.307124 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:10.307126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:10.307376 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:10.307624 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:10.307626 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:10.307627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:10.307697 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:10.308068 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:10.308071 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:10.308157 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.308268 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.308389 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
Sep 26 14:50:10 dist-test-slave-24mn krb5kdc[13633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434210, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:50:10 dist-test-slave-24mn krb5kdc[13633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434210, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:10.326081 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.319637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.12.68.254:41287)
0926 14:50:10.319730 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.319733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.319743 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:50:10.319769 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.319771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.319800 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.319857 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:10.320462 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.320911 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.321629 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.321836 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.321901 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.323893 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:10.323897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:10.323900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:10.323924 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:10.324922 (+ 998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.325258 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.325260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.325261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.325323 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.325506 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.325508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.325509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.325645 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:10.325734 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.325881 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.325955 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:50:10.328346 13677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:10.331089 13677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:10.343463 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.338178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:38989 (local address 127.0.0.1:60214)
0926 14:50:10.338574 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.338614 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.338742 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.339584 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.339590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.339614 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:10.339723 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.339729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.340921 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.340924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.342136 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.342146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.342995 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.342998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.343096 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.343130 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.343233 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.343339 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":67,"threads_started":1}
I20260926 14:50:10.343473 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.338271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60214 (local address 127.12.68.194:38989)
0926 14:50:10.338622 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.338633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.339065 (+ 432us) server_negotiation.cc:407] Connection header received
0926 14:50:10.339148 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.339153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.339345 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.339855 (+ 510us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:10.339998 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.340823 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.342238 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.342893 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.342984 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.343134 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.343262 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.343350 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":72,"threads_started":1}
I20260926 14:50:10.343798 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.338812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:33419 (local address 127.0.0.1:56910)
0926 14:50:10.339012 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.339025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.339095 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.340133 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.340135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.340153 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:10.340227 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.340233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.341415 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.341418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.342465 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.342475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.343431 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.343434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.343540 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.343566 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.343617 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.343676 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260926 14:50:10.344122 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.338276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:38303 (local address 127.0.0.1:48530)
0926 14:50:10.338750 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.338762 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.338844 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.339838 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.339840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.339857 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:10.339961 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.339967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.341246 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.341250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.342350 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.342357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.343455 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.343458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.343833 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.343855 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.343941 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.344005 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":283,"thread_start_us":89,"threads_started":1}
I20260926 14:50:10.344485 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.339415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.12.68.193:33419)
0926 14:50:10.339700 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.339711 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.339731 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:50:10.339796 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.339801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.339979 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.340154 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:10.340320 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.341324 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.342576 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.343308 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.344077 (+ 769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.344222 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.344271 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.344356 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":87,"threads_started":1}
I20260926 14:50:10.344506 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.339002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48530 (local address 127.12.68.195:38303)
0926 14:50:10.339431 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.339446 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.339467 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:50:10.339524 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.339529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.339701 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.339844 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:10.340064 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.341139 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.342444 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.343170 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.343267 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.343407 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.344319 (+ 912us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.344392 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":83,"threads_started":1}
I20260926 14:50:10.346272 13955 tablet_service.cc:1467] Processing CreateTablet for tablet 12ffa9d2583d4b88b0f6806032e30657 (DEFAULT_TABLE table=test-table [id=3984a5d862ab462cbc27d81ad725c27d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:10.347309 13955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ffa9d2583d4b88b0f6806032e30657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:10.346941 14095 tablet_service.cc:1467] Processing CreateTablet for tablet 12ffa9d2583d4b88b0f6806032e30657 (DEFAULT_TABLE table=test-table [id=3984a5d862ab462cbc27d81ad725c27d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:10.347865 14095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ffa9d2583d4b88b0f6806032e30657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:10.348466 13815 tablet_service.cc:1467] Processing CreateTablet for tablet 12ffa9d2583d4b88b0f6806032e30657 (DEFAULT_TABLE table=test-table [id=3984a5d862ab462cbc27d81ad725c27d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:10.349216 13815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ffa9d2583d4b88b0f6806032e30657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:10.354764 14184 tablet_bootstrap.cc:492] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0: Bootstrap starting.
I20260926 14:50:10.355736 14186 tablet_bootstrap.cc:492] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241: Bootstrap starting.
I20260926 14:50:10.357282 14184 tablet_bootstrap.cc:654] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:10.357688 14186 tablet_bootstrap.cc:654] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:10.358129 14184 log.cc:824] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:10.358438 14186 log.cc:824] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:10.358757 14185 tablet_bootstrap.cc:492] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd: Bootstrap starting.
I20260926 14:50:10.360337 14186 tablet_bootstrap.cc:492] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241: No bootstrap required, opened a new log
I20260926 14:50:10.360338 14184 tablet_bootstrap.cc:492] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0: No bootstrap required, opened a new log
I20260926 14:50:10.360503 14184 ts_tablet_manager.cc:1397] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260926 14:50:10.360505 14186 ts_tablet_manager.cc:1397] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:50:10.360721 14185 tablet_bootstrap.cc:654] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:10.361457 14185 log.cc:824] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:10.363205 14185 tablet_bootstrap.cc:492] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd: No bootstrap required, opened a new log
I20260926 14:50:10.363368 14185 ts_tablet_manager.cc:1397] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:50:10.363716 14184 raft_consensus.cc:348] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } }
I20260926 14:50:10.363933 14184 raft_consensus.cc:374] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:10.363983 14184 raft_consensus.cc:729] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a58b3da49ec4eafb0630b40bea6e7e0, State: Initialized, Role: FOLLOWER
I20260926 14:50:10.364001 14186 raft_consensus.cc:348] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } }
I20260926 14:50:10.364226 14186 raft_consensus.cc:374] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:10.364279 14186 raft_consensus.cc:729] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78497612d7c54fe49f891cf072745241, State: Initialized, Role: FOLLOWER
I20260926 14:50:10.364401 14184 consensus_queue.cc:260] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } }
I20260926 14:50:10.364681 14186 consensus_queue.cc:260] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } }
I20260926 14:50:10.365026 13881 heartbeater.cc:503] Master 127.12.68.254:41287 was elected leader, sending a full tablet report...
I20260926 14:50:10.365334 14184 ts_tablet_manager.cc:1428] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260926 14:50:10.365530 14186 ts_tablet_manager.cc:1428] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:50:10.365876 14161 heartbeater.cc:503] Master 127.12.68.254:41287 was elected leader, sending a full tablet report...
I20260926 14:50:10.366806 14185 raft_consensus.cc:348] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } }
I20260926 14:50:10.367028 14185 raft_consensus.cc:374] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:10.367079 14185 raft_consensus.cc:729] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98c3dce44e246f1bc6e32e4328a5cdd, State: Initialized, Role: FOLLOWER
I20260926 14:50:10.367420 14185 consensus_queue.cc:260] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } }
I20260926 14:50:10.367945 14021 heartbeater.cc:503] Master 127.12.68.254:41287 was elected leader, sending a full tablet report...
I20260926 14:50:10.368227 14185 ts_tablet_manager.cc:1428] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260926 14:50:10.404812 13882 tablet.cc:2369] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:10.535712 14192 raft_consensus.cc:482] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:10.535918 14192 raft_consensus.cc:504] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } }
I20260926 14:50:10.536966 14192 leader_election.cc:288] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78497612d7c54fe49f891cf072745241 (127.12.68.195:38303), 7a58b3da49ec4eafb0630b40bea6e7e0 (127.12.68.193:33419)
I20260926 14:50:10.540351 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.537239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:38303 (local address 127.12.68.194:36497)
0926 14:50:10.537360 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.537375 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.537456 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.537713 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.537716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.537734 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:10.537850 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.537856 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.538559 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.538563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.539338 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.539346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.539936 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.539940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.540077 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.540095 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.540156 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.540223 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":21}
I20260926 14:50:10.540371 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.537316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:36497 (local address 127.12.68.195:38303)
0926 14:50:10.537426 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.537430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.537445 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:50:10.537557 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.537560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.537607 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.537684 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:10.537962 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.538462 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.539457 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.539831 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.539949 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.540000 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.540185 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.540260 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:50:10.540916 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ffa9d2583d4b88b0f6806032e30657" candidate_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78497612d7c54fe49f891cf072745241" is_pre_election: true
I20260926 14:50:10.541178 14115 raft_consensus.cc:2454] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98c3dce44e246f1bc6e32e4328a5cdd in term 0.
I20260926 14:50:10.541513 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.537608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:39333 (local address 127.12.68.193:33419)
0926 14:50:10.537765 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.537770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.538188 (+ 418us) server_negotiation.cc:407] Connection header received
0926 14:50:10.538278 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.538282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.538322 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.538379 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:10.538645 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.539133 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.540582 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.540991 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.541089 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.541136 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.541344 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.541413 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 14:50:10.541536 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.537485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:33419 (local address 127.12.68.194:39333)
0926 14:50:10.538075 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.538088 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.538194 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.538403 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.538406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.538420 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:10.538537 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.538547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.539236 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.539239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.540442 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.540453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.541166 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.541170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.541241 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.541261 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.541329 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.541393 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":19,"thread_start_us":123,"threads_started":1}
I20260926 14:50:10.541591 13908 leader_election.cc:302] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78497612d7c54fe49f891cf072745241, b98c3dce44e246f1bc6e32e4328a5cdd; no voters:
I20260926 14:50:10.541878 14192 raft_consensus.cc:2781] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:10.541966 14192 raft_consensus.cc:482] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:10.542016 14192 raft_consensus.cc:3037] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:10.542284 13835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ffa9d2583d4b88b0f6806032e30657" candidate_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" is_pre_election: true
I20260926 14:50:10.542487 13835 raft_consensus.cc:2454] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98c3dce44e246f1bc6e32e4328a5cdd in term 0.
I20260926 14:50:10.543073 14192 raft_consensus.cc:504] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } }
I20260926 14:50:10.543336 14192 leader_election.cc:288] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [CANDIDATE]: Term 1 election: Requested vote from peers 78497612d7c54fe49f891cf072745241 (127.12.68.195:38303), 7a58b3da49ec4eafb0630b40bea6e7e0 (127.12.68.193:33419)
I20260926 14:50:10.543627 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ffa9d2583d4b88b0f6806032e30657" candidate_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78497612d7c54fe49f891cf072745241"
I20260926 14:50:10.543690 13835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ffa9d2583d4b88b0f6806032e30657" candidate_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0"
I20260926 14:50:10.543759 14115 raft_consensus.cc:3037] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:10.543787 13835 raft_consensus.cc:3037] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:10.544708 13835 raft_consensus.cc:2454] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98c3dce44e246f1bc6e32e4328a5cdd in term 1.
I20260926 14:50:10.544938 13908 leader_election.cc:302] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a58b3da49ec4eafb0630b40bea6e7e0, b98c3dce44e246f1bc6e32e4328a5cdd; no voters:
I20260926 14:50:10.544992 14115 raft_consensus.cc:2454] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98c3dce44e246f1bc6e32e4328a5cdd in term 1.
I20260926 14:50:10.545097 14192 raft_consensus.cc:2781] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:10.545276 14192 raft_consensus.cc:686] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [term 1 LEADER]: Becoming Leader. State: Replica: b98c3dce44e246f1bc6e32e4328a5cdd, State: Running, Role: LEADER
I20260926 14:50:10.545560 14192 consensus_queue.cc:237] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } }
W20260926 14:50:10.547286 14162 tablet.cc:2369] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:10.548537 13677 catalog_manager.cc:5495] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to b98c3dce44e246f1bc6e32e4328a5cdd (127.12.68.194). New cstate: current_term: 1 leader_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:10.581543 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.578133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.12.68.194:38989)
0926 14:50:10.578270 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.578274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.578287 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:10.578458 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.578460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.578512 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.578588 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:10.578827 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.579321 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.580220 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.580424 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.580507 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.580813 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:10.581176 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:10.581277 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.581366 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.581418 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 14:50:10.586577 13835 raft_consensus.cc:1261] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 1 FOLLOWER]: Refusing update from remote peer b98c3dce44e246f1bc6e32e4328a5cdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:10.586589 14115 raft_consensus.cc:1261] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241 [term 1 FOLLOWER]: Refusing update from remote peer b98c3dce44e246f1bc6e32e4328a5cdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:10.587157 14192 consensus_queue.cc:1035] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:10.587335 14194 consensus_queue.cc:1035] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:10.594376 14197 mvcc.cc:205] Tried to move back new op lower bound from 7333618526556614656 to 7333618526396973056. Current Snapshot: MvccSnapshot[applied={T|T < 7333618526556614656}]
I20260926 14:50:10.604548 13676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:10.604705 13676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:10.606856 13676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190: Sending DeleteTablet for 3 replicas of tablet 12ffa9d2583d4b88b0f6806032e30657
I20260926 14:50:10.607496 13955 tablet_service.cc:1514] Processing DeleteTablet for tablet 12ffa9d2583d4b88b0f6806032e30657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:10 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:60214
I20260926 14:50:10.607556 14095 tablet_service.cc:1514] Processing DeleteTablet for tablet 12ffa9d2583d4b88b0f6806032e30657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:10 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:48530
I20260926 14:50:10.607658 13815 tablet_service.cc:1514] Processing DeleteTablet for tablet 12ffa9d2583d4b88b0f6806032e30657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:10 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:56910
I20260926 14:50:10.607949 14206 tablet_replica.cc:330] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd: stopping tablet replica
I20260926 14:50:10.608250 14206 raft_consensus.cc:2229] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:10.608472 14207 tablet_replica.cc:330] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241: stopping tablet replica
I20260926 14:50:10.608718 14207 raft_consensus.cc:2229] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:10.608697 13676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 14:50:10.608997 14207 raft_consensus.cc:2258] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:50:10.609073 13676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:10.609761 14206 raft_consensus.cc:2258] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:10.610399 14207 ts_tablet_manager.cc:1905] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:10.613075 14207 ts_tablet_manager.cc:1918] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:10.613168 14207 log.cc:1196] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/12ffa9d2583d4b88b0f6806032e30657
I20260926 14:50:10.613327 13815 tablet_service.cc:1467] Processing CreateTablet for tablet 99f5aff2c4344ce9bcee39daae2bb66a (DEFAULT_TABLE table=test-table [id=2fa3a86a1fff4fbc9174a9bc8da89d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:10.613435 14207 ts_tablet_manager.cc:1939] T 12ffa9d2583d4b88b0f6806032e30657 P 78497612d7c54fe49f891cf072745241: Deleting consensus metadata
I20260926 14:50:10.613571 13815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f5aff2c4344ce9bcee39daae2bb66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:10.613528 13955 tablet_service.cc:1467] Processing CreateTablet for tablet 99f5aff2c4344ce9bcee39daae2bb66a (DEFAULT_TABLE table=test-table [id=2fa3a86a1fff4fbc9174a9bc8da89d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:10.613757 13955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f5aff2c4344ce9bcee39daae2bb66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:10.615526 14184 tablet_bootstrap.cc:492] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0: Bootstrap starting.
I20260926 14:50:10.616019 14185 tablet_bootstrap.cc:492] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd: Bootstrap starting.
I20260926 14:50:10.616441 14184 tablet_bootstrap.cc:654] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:10.616451 13664 catalog_manager.cc:4841] TS 78497612d7c54fe49f891cf072745241 (127.12.68.195:38303): tablet 12ffa9d2583d4b88b0f6806032e30657 (table test-table [id=3984a5d862ab462cbc27d81ad725c27d]) successfully deleted
I20260926 14:50:10.616853 14185 tablet_bootstrap.cc:654] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:10.616871 14095 tablet_service.cc:1467] Processing CreateTablet for tablet 99f5aff2c4344ce9bcee39daae2bb66a (DEFAULT_TABLE table=test-table [id=2fa3a86a1fff4fbc9174a9bc8da89d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:10.617084 14095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f5aff2c4344ce9bcee39daae2bb66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:10.617261 14206 ts_tablet_manager.cc:1905] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:10.617568 14184 tablet_bootstrap.cc:492] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0: No bootstrap required, opened a new log
I20260926 14:50:10.617642 14184 ts_tablet_manager.cc:1397] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:10.618010 14184 raft_consensus.cc:348] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.618095 14184 raft_consensus.cc:374] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:10.618124 14184 raft_consensus.cc:729] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a58b3da49ec4eafb0630b40bea6e7e0, State: Initialized, Role: FOLLOWER
I20260926 14:50:10.618233 14184 consensus_queue.cc:260] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.618468 14184 ts_tablet_manager.cc:1428] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:10.619266 14186 tablet_bootstrap.cc:492] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241: Bootstrap starting.
I20260926 14:50:10.619733 14206 ts_tablet_manager.cc:1918] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:10.619827 14206 log.cc:1196] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/12ffa9d2583d4b88b0f6806032e30657
I20260926 14:50:10.620119 14186 tablet_bootstrap.cc:654] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:10.620129 14206 ts_tablet_manager.cc:1939] T 12ffa9d2583d4b88b0f6806032e30657 P b98c3dce44e246f1bc6e32e4328a5cdd: Deleting consensus metadata
I20260926 14:50:10.621156 14186 tablet_bootstrap.cc:492] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241: No bootstrap required, opened a new log
I20260926 14:50:10.621135 13662 catalog_manager.cc:4841] TS b98c3dce44e246f1bc6e32e4328a5cdd (127.12.68.194:38989): tablet 12ffa9d2583d4b88b0f6806032e30657 (table test-table [id=3984a5d862ab462cbc27d81ad725c27d]) successfully deleted
I20260926 14:50:10.621243 14186 ts_tablet_manager.cc:1397] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:10.621546 14185 tablet_bootstrap.cc:492] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd: No bootstrap required, opened a new log
I20260926 14:50:10.621611 14185 ts_tablet_manager.cc:1397] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 14:50:10.621598 14186 raft_consensus.cc:348] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.621693 14186 raft_consensus.cc:374] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:10.621726 14186 raft_consensus.cc:729] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78497612d7c54fe49f891cf072745241, State: Initialized, Role: FOLLOWER
I20260926 14:50:10.621852 14186 consensus_queue.cc:260] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.621989 14185 raft_consensus.cc:348] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.622073 14185 raft_consensus.cc:374] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:10.622085 14186 ts_tablet_manager.cc:1428] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:10.622099 14185 raft_consensus.cc:729] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98c3dce44e246f1bc6e32e4328a5cdd, State: Initialized, Role: FOLLOWER
I20260926 14:50:10.622200 14185 consensus_queue.cc:260] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.622424 14185 ts_tablet_manager.cc:1428] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:10.624028 14208 tablet_replica.cc:330] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0: stopping tablet replica
I20260926 14:50:10.624289 14208 raft_consensus.cc:2229] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:10.624518 14208 raft_consensus.cc:2258] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:10.625792 14208 ts_tablet_manager.cc:1905] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:10.628414 14208 ts_tablet_manager.cc:1918] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:10.628500 14208 log.cc:1196] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/wal/wals/12ffa9d2583d4b88b0f6806032e30657
I20260926 14:50:10.628777 14208 ts_tablet_manager.cc:1939] T 12ffa9d2583d4b88b0f6806032e30657 P 7a58b3da49ec4eafb0630b40bea6e7e0: Deleting consensus metadata
I20260926 14:50:10.629741 13664 catalog_manager.cc:4841] TS 7a58b3da49ec4eafb0630b40bea6e7e0 (127.12.68.193:33419): tablet 12ffa9d2583d4b88b0f6806032e30657 (table test-table [id=3984a5d862ab462cbc27d81ad725c27d]) successfully deleted
I20260926 14:50:10.723198 14190 raft_consensus.cc:482] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:10.723356 14190 raft_consensus.cc:504] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.724668 14190 leader_election.cc:288] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b98c3dce44e246f1bc6e32e4328a5cdd (127.12.68.194:38989), 78497612d7c54fe49f891cf072745241 (127.12.68.195:38303)
I20260926 14:50:10.729269 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.725017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:36349 (local address 127.12.68.195:38303)
0926 14:50:10.725149 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.725154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.725368 (+ 214us) server_negotiation.cc:407] Connection header received
0926 14:50:10.725552 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.725555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.725602 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.725672 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:10.725971 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.726481 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.727870 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.728524 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.728644 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.728711 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.729057 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.729139 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:50:10.729431 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.725044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:35519 (local address 127.12.68.194:38989)
0926 14:50:10.725168 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.725173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.725435 (+ 262us) server_negotiation.cc:407] Connection header received
0926 14:50:10.725623 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.725626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.725672 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.725742 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:10.726065 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.726822 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.728139 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.728729 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.728824 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.728872 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.729243 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.729317 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:50:10.729846 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.724852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:38989 (local address 127.12.68.193:35519)
0926 14:50:10.725345 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.725359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.725473 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.725831 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.725833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.725846 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:10.725947 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.725954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.726943 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.726947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.728016 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.728024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.728979 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.728982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.729095 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.729116 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.729606 (+ 490us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.729686 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":208,"thread_start_us":57,"threads_started":1}
I20260926 14:50:10.730266 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.724850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:38303 (local address 127.12.68.193:36349)
0926 14:50:10.725277 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.725290 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.725408 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.725707 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.725710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.725727 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:10.725849 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.725857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.726584 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.726588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.727729 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.727739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.728775 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.728779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.728889 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.728914 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.729758 (+ 844us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.729839 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":81,"threads_started":1}
I20260926 14:50:10.730365 13966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f5aff2c4344ce9bcee39daae2bb66a" candidate_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" is_pre_election: true
I20260926 14:50:10.730587 13966 raft_consensus.cc:2454] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a58b3da49ec4eafb0630b40bea6e7e0 in term 0.
I20260926 14:50:10.731107 13766 leader_election.cc:302] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a58b3da49ec4eafb0630b40bea6e7e0, b98c3dce44e246f1bc6e32e4328a5cdd; no voters:
I20260926 14:50:10.731294 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f5aff2c4344ce9bcee39daae2bb66a" candidate_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78497612d7c54fe49f891cf072745241" is_pre_election: true
I20260926 14:50:10.731351 14190 raft_consensus.cc:2781] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:10.731419 14190 raft_consensus.cc:482] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:10.731423 14115 raft_consensus.cc:2454] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a58b3da49ec4eafb0630b40bea6e7e0 in term 0.
I20260926 14:50:10.731465 14190 raft_consensus.cc:3037] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:10.732717 14190 raft_consensus.cc:504] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.733069 14190 leader_election.cc:288] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [CANDIDATE]: Term 1 election: Requested vote from peers b98c3dce44e246f1bc6e32e4328a5cdd (127.12.68.194:38989), 78497612d7c54fe49f891cf072745241 (127.12.68.195:38303)
I20260926 14:50:10.733357 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f5aff2c4344ce9bcee39daae2bb66a" candidate_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78497612d7c54fe49f891cf072745241"
I20260926 14:50:10.733459 14115 raft_consensus.cc:3037] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:10.734438 14115 raft_consensus.cc:2454] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a58b3da49ec4eafb0630b40bea6e7e0 in term 1.
I20260926 14:50:10.734653 13966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f5aff2c4344ce9bcee39daae2bb66a" candidate_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd"
I20260926 14:50:10.734754 13966 raft_consensus.cc:3037] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:10.734740 13768 leader_election.cc:302] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78497612d7c54fe49f891cf072745241, 7a58b3da49ec4eafb0630b40bea6e7e0; no voters:
I20260926 14:50:10.734951 14190 raft_consensus.cc:2781] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:10.735131 14190 raft_consensus.cc:686] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a58b3da49ec4eafb0630b40bea6e7e0, State: Running, Role: LEADER
I20260926 14:50:10.735357 14190 consensus_queue.cc:237] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.735780 13966 raft_consensus.cc:2454] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a58b3da49ec4eafb0630b40bea6e7e0 in term 1.
I20260926 14:50:10.737524 13677 catalog_manager.cc:5495] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a58b3da49ec4eafb0630b40bea6e7e0 (127.12.68.193). New cstate: current_term: 1 leader_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } health_report { overall_health: UNKNOWN } } }
Sep 26 14:50:10 dist-test-slave-24mn krb5kdc[13633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:10.797379 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.789447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.12.68.254:41287)
0926 14:50:10.789566 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.789571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.789688 (+ 117us) server_negotiation.cc:407] Connection header received
0926 14:50:10.789803 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.789806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.789840 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.789901 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:10.790532 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.791038 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.791777 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.792011 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.792089 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.794205 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:10.794209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:10.794212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:10.794239 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:10.795748 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.796244 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.796247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.796249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.796324 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.796567 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.796570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.796572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.796737 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:10.796817 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.797105 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.797251 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":45}
I20260926 14:50:10.797379 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.789347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41287 (local address 127.0.0.1:41222)
0926 14:50:10.789607 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.789621 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.789721 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.789930 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.789933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.790335 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:10.790423 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.790429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.791126 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.791137 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.791674 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.791680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.792107 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.792111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.792196 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.792704 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:10.792715 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:10.794064 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:10.795856 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:10.795865 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:10.795872 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:10.796168 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:10.796418 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:10.796421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:10.796423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:10.796496 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:10.796851 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:10.796857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:10.796956 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.797049 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.797202 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
I20260926 14:50:10.802301 13677 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:41222:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 14:50:10.802973 13677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:10.807178 13955 tablet_service.cc:1467] Processing CreateTablet for tablet 25b67aef3ecf4492b822b39a22c8e41d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24d796bdbd4a450bb19986df0d1f6222]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:10.807174 13815 tablet_service.cc:1467] Processing CreateTablet for tablet 25b67aef3ecf4492b822b39a22c8e41d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24d796bdbd4a450bb19986df0d1f6222]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:10.807245 14095 tablet_service.cc:1467] Processing CreateTablet for tablet 25b67aef3ecf4492b822b39a22c8e41d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24d796bdbd4a450bb19986df0d1f6222]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:10.807538 13815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b67aef3ecf4492b822b39a22c8e41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:10.807538 13955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b67aef3ecf4492b822b39a22c8e41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:10.807588 14095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b67aef3ecf4492b822b39a22c8e41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:10.809918 14186 tablet_bootstrap.cc:492] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241: Bootstrap starting.
I20260926 14:50:10.810077 14185 tablet_bootstrap.cc:492] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd: Bootstrap starting.
I20260926 14:50:10.810137 14184 tablet_bootstrap.cc:492] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0: Bootstrap starting.
I20260926 14:50:10.810757 14186 tablet_bootstrap.cc:654] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:10.810859 14185 tablet_bootstrap.cc:654] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:10.810976 14184 tablet_bootstrap.cc:654] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:10.811789 14186 tablet_bootstrap.cc:492] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241: No bootstrap required, opened a new log
I20260926 14:50:10.811872 14186 ts_tablet_manager.cc:1397] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:10.812155 14184 tablet_bootstrap.cc:492] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0: No bootstrap required, opened a new log
I20260926 14:50:10.812197 14185 tablet_bootstrap.cc:492] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd: No bootstrap required, opened a new log
I20260926 14:50:10.812224 14184 ts_tablet_manager.cc:1397] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:10.812276 14185 ts_tablet_manager.cc:1397] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:10.812304 14186 raft_consensus.cc:348] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.812407 14186 raft_consensus.cc:374] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:10.812443 14186 raft_consensus.cc:729] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78497612d7c54fe49f891cf072745241, State: Initialized, Role: FOLLOWER
I20260926 14:50:10.812559 14186 consensus_queue.cc:260] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.812597 14184 raft_consensus.cc:348] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.812637 14185 raft_consensus.cc:348] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.812695 14184 raft_consensus.cc:374] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:10.812722 14185 raft_consensus.cc:374] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:10.812727 14184 raft_consensus.cc:729] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a58b3da49ec4eafb0630b40bea6e7e0, State: Initialized, Role: FOLLOWER
I20260926 14:50:10.812758 14185 raft_consensus.cc:729] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98c3dce44e246f1bc6e32e4328a5cdd, State: Initialized, Role: FOLLOWER
I20260926 14:50:10.812834 14186 ts_tablet_manager.cc:1428] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:10.812839 14184 consensus_queue.cc:260] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.812872 14185 consensus_queue.cc:260] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.813066 14184 ts_tablet_manager.cc:1428] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:10.813016 14191 tablet_replica.cc:439] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } }
I20260926 14:50:10.813103 14185 ts_tablet_manager.cc:1428] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:10.813133 14191 tablet_replica.cc:442] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:10.813232 14190 tablet_replica.cc:439] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } }
I20260926 14:50:10.813266 14192 tablet_replica.cc:439] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } }
I20260926 14:50:10.813339 14190 tablet_replica.cc:442] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:10.813365 14192 tablet_replica.cc:442] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:10.814982 14191 raft_consensus.cc:482] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:10.815061 14191 raft_consensus.cc:504] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.816017 14191 leader_election.cc:288] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a58b3da49ec4eafb0630b40bea6e7e0 (127.12.68.193:33419), b98c3dce44e246f1bc6e32e4328a5cdd (127.12.68.194:38989)
I20260926 14:50:10.819906 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.816502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:47193 (local address 127.12.68.194:38989)
0926 14:50:10.816615 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.816619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.816631 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:10.816722 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.816724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.816762 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.816828 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:10.817159 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.817707 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.818864 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.819437 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.819518 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.819553 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.819744 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.819798 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:50:10.820119 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.816250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:38989 (local address 127.12.68.195:47193)
0926 14:50:10.816549 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.816562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.816647 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.816919 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.816922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.816938 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:10.817038 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.817045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.817814 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.817817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.818771 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.818779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.819535 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.819539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.819635 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.819661 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.819736 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.819802 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":63,"threads_started":1}
I20260926 14:50:10.820458 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.816194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:50809 (local address 127.12.68.193:33419)
0926 14:50:10.816296 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.816300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.816534 (+ 234us) server_negotiation.cc:407] Connection header received
0926 14:50:10.816647 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.816650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.816691 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.816749 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:10.817054 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.817679 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.819324 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.819810 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.819947 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.819990 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.820284 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.820358 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:50:10.820511 13966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b67aef3ecf4492b822b39a22c8e41d" candidate_uuid: "78497612d7c54fe49f891cf072745241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" is_pre_election: true
I20260926 14:50:10.820631 13966 raft_consensus.cc:2454] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78497612d7c54fe49f891cf072745241 in term 0.
I20260926 14:50:10.820806 14226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.816150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:33419 (local address 127.12.68.195:50809)
0926 14:50:10.816441 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.816454 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.816554 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.816839 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.816842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.816857 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:10.816961 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.816967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.817778 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.817781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.819225 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.819235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.819928 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.819932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.820145 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.820168 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.820609 (+ 441us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.820681 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":59,"threads_started":1}
I20260926 14:50:10.820940 14046 leader_election.cc:302] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78497612d7c54fe49f891cf072745241, b98c3dce44e246f1bc6e32e4328a5cdd; no voters:
I20260926 14:50:10.821167 14191 raft_consensus.cc:2781] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:10.821220 14191 raft_consensus.cc:482] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:10.821203 13835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b67aef3ecf4492b822b39a22c8e41d" candidate_uuid: "78497612d7c54fe49f891cf072745241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" is_pre_election: true
I20260926 14:50:10.821264 14191 raft_consensus.cc:3037] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:10.821298 13835 raft_consensus.cc:2454] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78497612d7c54fe49f891cf072745241 in term 0.
I20260926 14:50:10.822180 14191 raft_consensus.cc:504] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.822458 14191 leader_election.cc:288] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [CANDIDATE]: Term 1 election: Requested vote from peers 7a58b3da49ec4eafb0630b40bea6e7e0 (127.12.68.193:33419), b98c3dce44e246f1bc6e32e4328a5cdd (127.12.68.194:38989)
I20260926 14:50:10.822750 13966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b67aef3ecf4492b822b39a22c8e41d" candidate_uuid: "78497612d7c54fe49f891cf072745241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd"
I20260926 14:50:10.822769 13835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b67aef3ecf4492b822b39a22c8e41d" candidate_uuid: "78497612d7c54fe49f891cf072745241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0"
I20260926 14:50:10.822841 13966 raft_consensus.cc:3037] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:10.822849 13835 raft_consensus.cc:3037] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:10.823619 13966 raft_consensus.cc:2454] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78497612d7c54fe49f891cf072745241 in term 1.
I20260926 14:50:10.823619 13835 raft_consensus.cc:2454] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78497612d7c54fe49f891cf072745241 in term 1.
I20260926 14:50:10.823869 14048 leader_election.cc:302] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78497612d7c54fe49f891cf072745241, 7a58b3da49ec4eafb0630b40bea6e7e0; no voters:
I20260926 14:50:10.824100 14191 raft_consensus.cc:2781] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:10.824258 14191 raft_consensus.cc:686] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [term 1 LEADER]: Becoming Leader. State: Replica: 78497612d7c54fe49f891cf072745241, State: Running, Role: LEADER
I20260926 14:50:10.824465 14191 consensus_queue.cc:237] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } }
I20260926 14:50:10.825212 14227 tablet_replica.cc:439] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241: TxnStatusTablet state changed. Reason: New leader 78497612d7c54fe49f891cf072745241. Latest consensus state: current_term: 1 leader_uuid: "78497612d7c54fe49f891cf072745241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } }
I20260926 14:50:10.825306 14227 tablet_replica.cc:442] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:10.825794 14229 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 14:50:10.825924 14229 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 14:50:10.826421 13678 catalog_manager.cc:5495] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78497612d7c54fe49f891cf072745241 (127.12.68.195). New cstate: current_term: 1 leader_uuid: "78497612d7c54fe49f891cf072745241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } health_report { overall_health: HEALTHY } } }
I20260926 14:50:10.838255 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.834024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.12.68.195:38303)
0926 14:50:10.834140 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.834144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.834157 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:10.834217 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.834220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.834275 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.834369 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:10.834566 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.835305 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.836473 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.836823 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.836921 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.837407 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:10.837859 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:10.837963 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.838066 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.838138 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:50:10.838255 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.833815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:38303 (local address 127.0.0.1:48544)
0926 14:50:10.833949 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.833964 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.834057 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.834384 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.834386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.834392 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:10.834472 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.834476 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.835405 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.835408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.836341 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.836352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.837047 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.837051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.837208 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.837276 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:10.837955 (+ 679us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:10.837970 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.838044 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.838124 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 14:50:10.840575 13835 raft_consensus.cc:1261] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 1 FOLLOWER]: Refusing update from remote peer 78497612d7c54fe49f891cf072745241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:10.840772 13966 raft_consensus.cc:1261] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd [term 1 FOLLOWER]: Refusing update from remote peer 78497612d7c54fe49f891cf072745241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:10.840953 14227 consensus_queue.cc:1035] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:10.841244 14191 consensus_queue.cc:1035] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:10.842257 14190 tablet_replica.cc:439] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0: TxnStatusTablet state changed. Reason: New leader 78497612d7c54fe49f891cf072745241. Latest consensus state: current_term: 1 leader_uuid: "78497612d7c54fe49f891cf072745241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } }
I20260926 14:50:10.842355 14190 tablet_replica.cc:442] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:10.842548 14192 tablet_replica.cc:439] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd: TxnStatusTablet state changed. Reason: New leader 78497612d7c54fe49f891cf072745241. Latest consensus state: current_term: 1 leader_uuid: "78497612d7c54fe49f891cf072745241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } }
I20260926 14:50:10.842640 14192 tablet_replica.cc:442] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:10.844091 14191 tablet_replica.cc:439] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78497612d7c54fe49f891cf072745241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } }
I20260926 14:50:10.844197 14191 tablet_replica.cc:442] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:10.845064 14192 tablet_replica.cc:439] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78497612d7c54fe49f891cf072745241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } }
I20260926 14:50:10.845153 14192 tablet_replica.cc:442] T 25b67aef3ecf4492b822b39a22c8e41d P b98c3dce44e246f1bc6e32e4328a5cdd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:10.847138 14227 tablet_replica.cc:439] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78497612d7c54fe49f891cf072745241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } }
I20260926 14:50:10.847231 14227 tablet_replica.cc:442] T 25b67aef3ecf4492b822b39a22c8e41d P 78497612d7c54fe49f891cf072745241: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:10.847661 14198 mvcc.cc:205] Tried to move back new op lower bound from 7333618527599026176 to 7333618527538872320. Current Snapshot: MvccSnapshot[applied={T|T < 7333618527599026176}]
I20260926 14:50:10.847817 14190 tablet_replica.cc:439] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78497612d7c54fe49f891cf072745241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a58b3da49ec4eafb0630b40bea6e7e0" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33419 } } peers { permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 } } peers { permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 } } }
I20260926 14:50:10.847927 14190 tablet_replica.cc:442] T 25b67aef3ecf4492b822b39a22c8e41d P 7a58b3da49ec4eafb0630b40bea6e7e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:10.852219 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.848640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.12.68.193:33419)
0926 14:50:10.849289 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.849292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.849301 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:10.849350 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.849352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.849394 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.849471 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:10.849697 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.850198 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.850977 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.851190 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.851270 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.851510 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:10.851871 (+ 361us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:10.852003 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.852061 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.852109 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587}
I20260926 14:50:10.861936 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.858696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:36197 (local address 127.12.68.195:38303)
0926 14:50:10.858832 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.858835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.858924 (+ 89us) server_negotiation.cc:407] Connection header received
0926 14:50:10.859037 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.859039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.859077 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.859133 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:10.859374 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.859840 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.861048 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.861270 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.861344 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.861612 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:10.861676 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:10.861721 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.861809 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.861848 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 14:50:10.862146 14022 tablet.cc:2369] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:10.862275 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.858574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:38303 (local address 127.12.68.193:36197)
0926 14:50:10.858851 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.858864 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.858958 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.859151 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.859154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.859160 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:10.859263 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.859269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.859968 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.859971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.860931 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.860941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.861361 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.861364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.861456 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.861502 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:10.861740 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:10.861745 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.861803 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.861873 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":94,"threads_started":1}
I20260926 14:50:10.868290 14115 raft_consensus.cc:1261] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241 [term 1 FOLLOWER]: Refusing update from remote peer 7a58b3da49ec4eafb0630b40bea6e7e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:10.868289 13966 raft_consensus.cc:1261] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd [term 1 FOLLOWER]: Refusing update from remote peer 7a58b3da49ec4eafb0630b40bea6e7e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:10.868710 14214 consensus_queue.cc:1035] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78497612d7c54fe49f891cf072745241" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:10.868853 14190 consensus_queue.cc:1035] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98c3dce44e246f1bc6e32e4328a5cdd" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:10.873057 14199 mvcc.cc:205] Tried to move back new op lower bound from 7333618527712579584 to 7333618527174610944. Current Snapshot: MvccSnapshot[applied={T|T < 7333618527712579584}]
I20260926 14:50:10.889595 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.885928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:33419 (local address 127.12.68.195:54501)
0926 14:50:10.886248 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.886260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.886341 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.886532 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.886534 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.886538 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:10.886631 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.886635 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.887559 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.887563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.888404 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.888415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.888960 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.888962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.889046 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.889084 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:10.889326 (+ 242us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:10.889330 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.889371 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.889442 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":94,"threads_started":1}
I20260926 14:50:10.889621 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.886014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:54501 (local address 127.12.68.193:33419)
0926 14:50:10.886120 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.886124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.886319 (+ 195us) server_negotiation.cc:407] Connection header received
0926 14:50:10.886405 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.886408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.886452 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.886517 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:10.886713 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.887463 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.888506 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.888855 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.888952 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.889185 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:10.889258 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:10.889322 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.889400 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.889480 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:50:10.914489 13676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:10.914644 13676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:10.916507 13676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c5b9f00e888450e84d24b825e888190: Sending DeleteTablet for 3 replicas of tablet 99f5aff2c4344ce9bcee39daae2bb66a
I20260926 14:50:10.917017 13955 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f5aff2c4344ce9bcee39daae2bb66a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:10 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:60214
I20260926 14:50:10.917155 13815 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f5aff2c4344ce9bcee39daae2bb66a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:10 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:56910
I20260926 14:50:10.917193 14206 tablet_replica.cc:330] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd: stopping tablet replica
I20260926 14:50:10.917300 14206 raft_consensus.cc:2229] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:10.917407 14206 raft_consensus.cc:2258] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:10.917563 14208 tablet_replica.cc:330] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0: stopping tablet replica
I20260926 14:50:10.917595 14095 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f5aff2c4344ce9bcee39daae2bb66a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:10 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:48530
I20260926 14:50:10.917665 14208 raft_consensus.cc:2229] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:10.917737 14207 tablet_replica.cc:330] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241: stopping tablet replica
I20260926 14:50:10.917824 14207 raft_consensus.cc:2229] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:10.917913 14207 raft_consensus.cc:2258] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:10.917907 14208 raft_consensus.cc:2258] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:10.918623 14206 ts_tablet_manager.cc:1905] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:10.919687 14207 ts_tablet_manager.cc:1905] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:10.921221 14206 ts_tablet_manager.cc:1918] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:10.921291 14206 log.cc:1196] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/99f5aff2c4344ce9bcee39daae2bb66a
I20260926 14:50:10.921512 14206 ts_tablet_manager.cc:1939] T 99f5aff2c4344ce9bcee39daae2bb66a P b98c3dce44e246f1bc6e32e4328a5cdd: Deleting consensus metadata
I20260926 14:50:10.922250 14207 ts_tablet_manager.cc:1918] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:10.922308 14207 log.cc:1196] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/99f5aff2c4344ce9bcee39daae2bb66a
I20260926 14:50:10.922374 14208 ts_tablet_manager.cc:1905] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:10.922520 14207 ts_tablet_manager.cc:1939] T 99f5aff2c4344ce9bcee39daae2bb66a P 78497612d7c54fe49f891cf072745241: Deleting consensus metadata
I20260926 14:50:10.923190 13664 catalog_manager.cc:4841] TS 78497612d7c54fe49f891cf072745241 (127.12.68.195:38303): tablet 99f5aff2c4344ce9bcee39daae2bb66a (table test-table [id=2fa3a86a1fff4fbc9174a9bc8da89d11]) successfully deleted
I20260926 14:50:10.924723 14208 ts_tablet_manager.cc:1918] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:10.924777 14208 log.cc:1196] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434204424021-12563-0/minicluster-data/ts-0/wal/wals/99f5aff2c4344ce9bcee39daae2bb66a
I20260926 14:50:10.924978 14208 ts_tablet_manager.cc:1939] T 99f5aff2c4344ce9bcee39daae2bb66a P 7a58b3da49ec4eafb0630b40bea6e7e0: Deleting consensus metadata
I20260926 14:50:10.932060 13662 catalog_manager.cc:4841] TS b98c3dce44e246f1bc6e32e4328a5cdd (127.12.68.194:38989): tablet 99f5aff2c4344ce9bcee39daae2bb66a (table test-table [id=2fa3a86a1fff4fbc9174a9bc8da89d11]) successfully deleted
I20260926 14:50:10.932301 13664 catalog_manager.cc:4841] TS 7a58b3da49ec4eafb0630b40bea6e7e0 (127.12.68.193:33419): tablet 99f5aff2c4344ce9bcee39daae2bb66a (table test-table [id=2fa3a86a1fff4fbc9174a9bc8da89d11]) successfully deleted
Sep 26 14:50:10 dist-test-slave-24mn krb5kdc[13633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434210, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.68.193@KRBTEST.COM
I20260926 14:50:10.947176 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.919410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.12.68.193:33419)
0926 14:50:10.923081 (+ 3671us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.923085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.936143 (+ 13058us) server_negotiation.cc:407] Connection header received
0926 14:50:10.936291 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.936294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.936340 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.936417 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:10.937309 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.938124 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.939319 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.939668 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.939768 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.943288 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:10.943297 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:10.943312 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:10.943337 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:10.945238 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.945753 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.945759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.945761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.945986 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.946239 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.946242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.946252 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.946516 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:10.946722 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.946902 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.947011 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3599}
W20260926 14:50:10.947578 13875 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56930
I20260926 14:50:10.957787 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.949381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.12.68.254:41287)
0926 14:50:10.949489 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.949493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.949896 (+ 403us) server_negotiation.cc:407] Connection header received
0926 14:50:10.949986 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.949990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.950038 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.950115 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:10.950833 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.951615 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.952720 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.953060 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.953157 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.954617 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:10.954620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:10.954623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:10.954649 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:10.956189 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.956622 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.956625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.956627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.956712 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:10.956962 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:10.956965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:10.956967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:10.957158 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:10.957279 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.957521 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.957624 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 14:50:10.958292 13676 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41234
I20260926 14:50:10.959620 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 13750
I20260926 14:50:10.967089 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 13890
I20260926 14:50:10.972813 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 14030
I20260926 14:50:10.977977 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 13646
2026-09-26T14:50:10Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2546 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:11 dist-test-slave-24mn krb5kdc[14251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:11 dist-test-slave-24mn krb5kdc[14251](info): set up 2 sockets
Sep 26 14:50:11 dist-test-slave-24mn krb5kdc[14251](info): commencing operation
krb5kdc: starting...
W20260926 14:50:11.927434 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:50:11 dist-test-slave-24mn krb5kdc[14251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:50:12.032658 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:41443
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:41443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:12.134733 14264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:12.134946 14264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:12.134987 14264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:12.138899 14264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:12.138944 14264 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 14:50:12.138960 14264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:12.138991 14264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:12.139020 14264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:12.139045 14264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:12.143980 14264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:41443
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:41443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:12.144840 14264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:12.145740 14264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:12.153635 14269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:12.153704 14270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:12.153697 14272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:12.153774 14264 server_base.cc:1034] running on GCE node
I20260926 14:50:12.154395 14264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:12.155401 14264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:12.156548 14264 hybrid_clock.cc:648] HybridClock initialized: now 1790434212156532 us; error 27 us; skew 500 ppm
Sep 26 14:50:12 dist-test-slave-24mn krb5kdc[14251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:12.159013 14264 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:50:12.159794 14264 webserver.cc:460] Webserver started at http://127.12.68.254:38113/ using document root <none> and password file <none>
I20260926 14:50:12.160308 14264 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:12.160372 14264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:12.160555 14264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:12.162452 14264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "6f96d6cdeaaf4007a065b434aafa482f"
format_stamp: "Formatted at 2026-09-26 14:50:12 on dist-test-slave-24mn"
server_key: "7573aa7e1ec31e035bb9b449820866ac"
server_key_iv: "ef1cd5b21a2cc51a2346dd8344681ffa"
server_key_version: "encryptionkey@0"
I20260926 14:50:12.162949 14264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "6f96d6cdeaaf4007a065b434aafa482f"
format_stamp: "Formatted at 2026-09-26 14:50:12 on dist-test-slave-24mn"
server_key: "7573aa7e1ec31e035bb9b449820866ac"
server_key_iv: "ef1cd5b21a2cc51a2346dd8344681ffa"
server_key_version: "encryptionkey@0"
I20260926 14:50:12.166357 14264 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:50:12.168514 14279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:12.169617 14264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:12.169744 14264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "6f96d6cdeaaf4007a065b434aafa482f"
format_stamp: "Formatted at 2026-09-26 14:50:12 on dist-test-slave-24mn"
server_key: "7573aa7e1ec31e035bb9b449820866ac"
server_key_iv: "ef1cd5b21a2cc51a2346dd8344681ffa"
server_key_version: "encryptionkey@0"
I20260926 14:50:12.169829 14264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:12.190851 14264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:12.193380 14264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:12.193537 14264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:12.200485 14264 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:41443
I20260926 14:50:12.200493 14341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:41443 every 8 connection(s)
I20260926 14:50:12.201308 14264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:12.203815 14342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:12.207262 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 14264
I20260926 14:50:12.207369 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:12.207638 12563 external_mini_cluster.cc:1428] Setting key 5f59805434e9342971939e63a8224c86
I20260926 14:50:12.209414 14342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f: Bootstrap starting.
I20260926 14:50:12.211871 14342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:12.212692 14342 log.cc:824] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:12.214637 14342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f: No bootstrap required, opened a new log
Sep 26 14:50:12 dist-test-slave-24mn krb5kdc[14251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:12.218410 14342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f96d6cdeaaf4007a065b434aafa482f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41443 } }
I20260926 14:50:12.218655 14342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:12.218711 14342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f96d6cdeaaf4007a065b434aafa482f, State: Initialized, Role: FOLLOWER
I20260926 14:50:12.219165 14342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f96d6cdeaaf4007a065b434aafa482f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41443 } }
I20260926 14:50:12.219290 14342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:12.219355 14342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:12.219447 14342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:12.220238 14342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f96d6cdeaaf4007a065b434aafa482f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41443 } }
I20260926 14:50:12.220542 14342 leader_election.cc:302] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f96d6cdeaaf4007a065b434aafa482f; no voters:
I20260926 14:50:12.220800 14342 leader_election.cc:288] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:12.221006 14347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:12.221179 14347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [term 1 LEADER]: Becoming Leader. State: Replica: 6f96d6cdeaaf4007a065b434aafa482f, State: Running, Role: LEADER
I20260926 14:50:12.221455 14347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f96d6cdeaaf4007a065b434aafa482f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41443 } }
I20260926 14:50:12.221864 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.208862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.12.68.254:41443)
0926 14:50:12.209249 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:12.209266 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:12.209306 (+ 40us) server_negotiation.cc:407] Connection header received
0926 14:50:12.210088 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:12.210110 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:12.210493 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:12.210841 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:12.211427 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.212904 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.213847 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.214512 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.214672 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.217117 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:12.217124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:12.217140 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:12.217168 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:12.218958 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.219393 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.219399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.219405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.219532 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.219797 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.219801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.219802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.220222 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:12.220396 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:12.220655 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 14:50:12.221053 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260926 14:50:12.222002 14342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:12.222568 14347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f96d6cdeaaf4007a065b434aafa482f. Latest consensus state: current_term: 1 leader_uuid: "6f96d6cdeaaf4007a065b434aafa482f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f96d6cdeaaf4007a065b434aafa482f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41443 } } }
I20260926 14:50:12.222677 14347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:12.222910 14348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f96d6cdeaaf4007a065b434aafa482f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f96d6cdeaaf4007a065b434aafa482f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41443 } } }
I20260926 14:50:12.222997 14348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f [sys.catalog]: This master's current role is: LEADER
W20260926 14:50:12.226413 14362 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:50:12.226475 14362 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:50:12.226544 14353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:12.227212 14353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:12.232263 14353 catalog_manager.cc:1347] Generated new cluster ID: da196098e97a4b4aaea56690418c0fb1
I20260926 14:50:12.232318 14353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:12.245153 14353 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:12.245965 14353 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:12.252036 14353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f: Generated new TSK 0
I20260926 14:50:12.252604 14353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:50:12.280859 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:41443
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:12.384680 14368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:12.384893 14368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:12.384934 14368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:12.388763 14368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:12.388808 14368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:12.388849 14368 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 14:50:12.388875 14368 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:12.388922 14368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:12.393817 14368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.68.254:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:12.394655 14368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:12.395531 14368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:50:12.404170 14368 server_base.cc:1034] running on GCE node
W20260926 14:50:12.404184 14373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:12.404321 14376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:12.404184 14374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:12.404729 14368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:12.405284 14368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:12.406453 14368 hybrid_clock.cc:648] HybridClock initialized: now 1790434212406430 us; error 33 us; skew 500 ppm
Sep 26 14:50:12 dist-test-slave-24mn krb5kdc[14251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:12.409046 14368 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:50:12.409859 14368 webserver.cc:460] Webserver started at http://127.12.68.193:43215/ using document root <none> and password file <none>
I20260926 14:50:12.410346 14368 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:12.410403 14368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:12.410588 14368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:12.412456 14368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "92a259b4b95e4261b7e86713bbf992d7"
format_stamp: "Formatted at 2026-09-26 14:50:12 on dist-test-slave-24mn"
server_key: "5734f3753cf6f2bac3d3c74a5b9a42fc"
server_key_iv: "5ebfe3a7efefca52d7d126194eb55f37"
server_key_version: "encryptionkey@0"
I20260926 14:50:12.412928 14368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "92a259b4b95e4261b7e86713bbf992d7"
format_stamp: "Formatted at 2026-09-26 14:50:12 on dist-test-slave-24mn"
server_key: "5734f3753cf6f2bac3d3c74a5b9a42fc"
server_key_iv: "5ebfe3a7efefca52d7d126194eb55f37"
server_key_version: "encryptionkey@0"
I20260926 14:50:12.416319 14368 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:12.418366 14383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:12.419528 14368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:12.419639 14368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "92a259b4b95e4261b7e86713bbf992d7"
format_stamp: "Formatted at 2026-09-26 14:50:12 on dist-test-slave-24mn"
server_key: "5734f3753cf6f2bac3d3c74a5b9a42fc"
server_key_iv: "5ebfe3a7efefca52d7d126194eb55f37"
server_key_version: "encryptionkey@0"
I20260926 14:50:12.419730 14368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:12.428267 14368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:12.430871 14368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:12.431015 14368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:12.432283 14368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:12.432329 14368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:12.432375 14368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:12.432401 14368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:50:12 dist-test-slave-24mn krb5kdc[14251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:12.441922 14368 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:44695
I20260926 14:50:12.441967 14498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:44695 every 8 connection(s)
I20260926 14:50:12.442817 14368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:12.444798 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.432864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:33047 (local address 127.12.68.254:41443)
0926 14:50:12.432975 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:12.432980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:12.434434 (+ 1454us) server_negotiation.cc:407] Connection header received
0926 14:50:12.435503 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:12.435506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:12.435572 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:12.435640 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:12.436711 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.437323 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.438926 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.439231 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.439317 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.441866 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:12.441870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:12.441873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:12.441897 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:12.443159 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.443645 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.443648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.443649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.443727 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.443988 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.443990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.443992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.444158 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:12.444232 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:12.444591 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 14:50:12.444685 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:50:12.445575 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 14368
I20260926 14:50:12.445689 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:12.445701 14392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.433814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41443 (local address 127.12.68.193:33047)
0926 14:50:12.434314 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:12.434351 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:12.435317 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:12.435803 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:12.435811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:12.436175 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:12.436575 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.436587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.437439 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.437443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.438800 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.438812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.439380 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.439385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.439573 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.440092 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:12.440112 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:12.441697 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:12.443279 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:12.443284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:12.443296 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:12.443570 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:12.443790 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:12.443792 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:12.443793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:12.443930 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:12.444229 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:12.444234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:12.444417 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 14:50:12.444544 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 14:50:12.444817 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":123,"threads_started":1}
I20260926 14:50:12.445955 12563 external_mini_cluster.cc:1428] Setting key 7d1ed95f16dcd890e9f9ed6071b068d6
I20260926 14:50:12.446755 14499 heartbeater.cc:347] Connected to a master server at 127.12.68.254:41443
I20260926 14:50:12.446942 14499 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:12.447352 14499 heartbeater.cc:511] Master 127.12.68.254:41443 requested a full tablet report, sending...
I20260926 14:50:12.449419 14295 ts_manager.cc:194] Registered new tserver with Master: 92a259b4b95e4261b7e86713bbf992d7 (127.12.68.193:44695)
I20260926 14:50:12.450445 14295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:33047
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
I20260926 14:50:12.457811 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.451164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:44353 (local address 127.12.68.254:41443)
0926 14:50:12.451268 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:12.451272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:12.451477 (+ 205us) server_negotiation.cc:407] Connection header received
0926 14:50:12.451567 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:12.451570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:12.451613 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:12.451703 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:12.452167 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.452909 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.453851 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.454187 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.454295 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.455140 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:12.455144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:12.455147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:12.455173 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:12.456328 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.456665 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.456667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.456668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.456732 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.457025 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.457028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.457030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.457233 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:12.457323 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:12.457576 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 14:50:12.457670 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:50:12.457820 14506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.451102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41443 (local address 127.12.68.193:44353)
0926 14:50:12.451397 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:12.451410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:12.451498 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:12.451717 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:12.451720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:12.451935 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:12.452040 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.452047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.453036 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.453039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.453729 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.453738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.454286 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.454288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.454349 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.454732 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:12.454742 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:12.455009 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:12.456416 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:12.456418 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:12.456420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:12.456591 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:12.456827 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:12.456830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:12.456831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:12.456913 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:12.457331 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:12.457334 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:12.457409 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 14:50:12.457505 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 14:50:12.457644 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:50:12.470378 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:41443
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:12.572639 14508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:12.572888 14508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:12.572952 14508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:12.576993 14508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:12.577041 14508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:12.577077 14508 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 14:50:12.577100 14508 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:12.577134 14508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:12.582031 14508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.68.254:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:12.582883 14508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:12.583748 14508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:12.592554 14514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:12.592553 14513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:12.592545 14516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:12.592727 14508 server_base.cc:1034] running on GCE node
I20260926 14:50:12.593147 14508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:12.593694 14508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:12.594852 14508 hybrid_clock.cc:648] HybridClock initialized: now 1790434212594837 us; error 30 us; skew 500 ppm
Sep 26 14:50:12 dist-test-slave-24mn krb5kdc[14251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:12.597307 14508 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:50:12.598011 14508 webserver.cc:460] Webserver started at http://127.12.68.194:42763/ using document root <none> and password file <none>
I20260926 14:50:12.598464 14508 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:12.598515 14508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:12.598683 14508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:12.600570 14508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "e7430a561f644e57888513349abf504c"
format_stamp: "Formatted at 2026-09-26 14:50:12 on dist-test-slave-24mn"
server_key: "2d092be234aaaad19d0cad61a0daf05e"
server_key_iv: "e2f4b910e3d9019ae5bcc67431299cbd"
server_key_version: "encryptionkey@0"
I20260926 14:50:12.601120 14508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "e7430a561f644e57888513349abf504c"
format_stamp: "Formatted at 2026-09-26 14:50:12 on dist-test-slave-24mn"
server_key: "2d092be234aaaad19d0cad61a0daf05e"
server_key_iv: "e2f4b910e3d9019ae5bcc67431299cbd"
server_key_version: "encryptionkey@0"
I20260926 14:50:12.604357 14508 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:12.606451 14523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:12.607511 14508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:12.607596 14508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "e7430a561f644e57888513349abf504c"
format_stamp: "Formatted at 2026-09-26 14:50:12 on dist-test-slave-24mn"
server_key: "2d092be234aaaad19d0cad61a0daf05e"
server_key_iv: "e2f4b910e3d9019ae5bcc67431299cbd"
server_key_version: "encryptionkey@0"
I20260926 14:50:12.607685 14508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:12.623827 14508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:12.626541 14508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:12.626688 14508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:12.627950 14508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:12.627996 14508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:12.628052 14508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:12.628077 14508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:50:12 dist-test-slave-24mn krb5kdc[14251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:12.636929 14508 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:43401
I20260926 14:50:12.636978 14638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:43401 every 8 connection(s)
I20260926 14:50:12.637787 14508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:12.640285 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.628633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:51865 (local address 127.12.68.254:41443)
0926 14:50:12.628777 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:12.628783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:12.629572 (+ 789us) server_negotiation.cc:407] Connection header received
0926 14:50:12.630534 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:12.630537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:12.630594 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:12.630662 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:12.632144 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.632704 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.634272 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.634611 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.634727 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.637405 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:12.637408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:12.637410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:12.637433 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:12.638763 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.639166 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.639168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.639170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.639242 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.639465 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.639467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.639468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.639621 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:12.639684 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:12.640085 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 14:50:12.640173 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:50:12.640929 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.628994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41443 (local address 127.12.68.194:51865)
0926 14:50:12.629411 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:12.629456 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:12.630298 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:12.631034 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:12.631042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:12.631403 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:12.632005 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.632024 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.632810 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.632813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.634126 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.634137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.634781 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.634784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.634953 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.635503 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:12.635522 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:12.637253 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:12.638856 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:12.638860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:12.638869 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:12.639087 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:12.639306 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:12.639308 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:12.639309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:12.639413 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:12.639689 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:12.639693 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:12.639874 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 14:50:12.640056 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 14:50:12.640263 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
I20260926 14:50:12.641911 14639 heartbeater.cc:347] Connected to a master server at 127.12.68.254:41443
I20260926 14:50:12.642022 14639 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:12.642416 14639 heartbeater.cc:511] Master 127.12.68.254:41443 requested a full tablet report, sending...
I20260926 14:50:12.643374 14296 ts_manager.cc:194] Registered new tserver with Master: e7430a561f644e57888513349abf504c (127.12.68.194:43401)
I20260926 14:50:12.643950 14296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:51865
I20260926 14:50:12.645227 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 14508
I20260926 14:50:12.645321 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:12.645576 12563 external_mini_cluster.cc:1428] Setting key 072301c81e8080fbb726874b8af0da74
I20260926 14:50:12.650332 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.643868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41443 (local address 127.12.68.194:39993)
0926 14:50:12.644101 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:12.644113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:12.644200 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:12.644393 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:12.644395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:12.644534 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:12.644665 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.644671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.645616 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.645619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.646250 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.646259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.646672 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.646675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.646758 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.647126 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:12.647135 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:12.647404 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:12.649051 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:12.649053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:12.649054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:12.649219 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:12.649456 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:12.649458 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:12.649459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:12.649525 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:12.649939 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:12.649942 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:12.650015 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 14:50:12.650124 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 14:50:12.650218 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260926 14:50:12.650408 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.643962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:39993 (local address 127.12.68.254:41443)
0926 14:50:12.644064 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:12.644067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:12.644168 (+ 101us) server_negotiation.cc:407] Connection header received
0926 14:50:12.644270 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:12.644273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:12.644312 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:12.644386 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:12.644764 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.645469 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.646356 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.646575 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.646656 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.647518 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:12.647522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:12.647524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:12.647546 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:12.648932 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.649302 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.649305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.649306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.649381 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.649592 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.649594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.649596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.649821 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:12.649912 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:12.650170 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 14:50:12.650274 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:50:12.668921 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:41443
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:12.771587 14648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:12.771811 14648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:12.771852 14648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:12.775785 14648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:12.775835 14648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:12.775871 14648 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 14:50:12.775899 14648 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:12.775933 14648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:12.780864 14648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.68.254:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:12.781745 14648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:12.782637 14648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:12.791286 14656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:12.791245 14654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:12.791379 14653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:12.791433 14648 server_base.cc:1034] running on GCE node
I20260926 14:50:12.791863 14648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:12.792480 14648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:12.793639 14648 hybrid_clock.cc:648] HybridClock initialized: now 1790434212793615 us; error 42 us; skew 500 ppm
Sep 26 14:50:12 dist-test-slave-24mn krb5kdc[14251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:12.796294 14648 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:50:12.797219 14648 webserver.cc:460] Webserver started at http://127.12.68.195:34115/ using document root <none> and password file <none>
I20260926 14:50:12.797842 14648 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:12.797962 14648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:12.798238 14648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:12.800650 14648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "e4709deb2fde426091fdbda06ee64cfc"
format_stamp: "Formatted at 2026-09-26 14:50:12 on dist-test-slave-24mn"
server_key: "10fe59bfbcdacde4a6e17f7abd2844b2"
server_key_iv: "83719c1068a375fb3354a005799aaf0c"
server_key_version: "encryptionkey@0"
I20260926 14:50:12.801208 14648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "e4709deb2fde426091fdbda06ee64cfc"
format_stamp: "Formatted at 2026-09-26 14:50:12 on dist-test-slave-24mn"
server_key: "10fe59bfbcdacde4a6e17f7abd2844b2"
server_key_iv: "83719c1068a375fb3354a005799aaf0c"
server_key_version: "encryptionkey@0"
I20260926 14:50:12.804652 14648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:50:12.806738 14663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:12.807823 14648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:50:12.807933 14648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "e4709deb2fde426091fdbda06ee64cfc"
format_stamp: "Formatted at 2026-09-26 14:50:12 on dist-test-slave-24mn"
server_key: "10fe59bfbcdacde4a6e17f7abd2844b2"
server_key_iv: "83719c1068a375fb3354a005799aaf0c"
server_key_version: "encryptionkey@0"
I20260926 14:50:12.808018 14648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:12.830458 14648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:12.833287 14648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:12.833437 14648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:12.834718 14648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:12.834771 14648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:12.834821 14648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:12.834848 14648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:50:12 dist-test-slave-24mn krb5kdc[14251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:12.844467 14648 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:39993
I20260926 14:50:12.844708 14778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:39993 every 8 connection(s)
I20260926 14:50:12.845315 14648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:12.847067 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.835359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:55097 (local address 127.12.68.254:41443)
0926 14:50:12.835479 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:12.835484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:12.836961 (+ 1477us) server_negotiation.cc:407] Connection header received
0926 14:50:12.838031 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:12.838039 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:12.838099 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:12.838185 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:12.839242 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.839826 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.840984 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.841220 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.841311 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.843646 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:12.843650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:12.843653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:12.843678 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:12.845096 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.845558 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.845562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.845564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.845651 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.846137 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.846140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.846141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.846363 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:12.846443 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:12.846862 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 14:50:12.846953 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:50:12.847743 14672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.836350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41443 (local address 127.12.68.195:55097)
0926 14:50:12.836835 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:12.836874 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:12.837813 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:12.838339 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:12.838345 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:12.838676 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:12.839099 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.839111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.839958 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.839962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.840870 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.840879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.841362 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.841366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.841515 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.842033 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:12.842053 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:12.843497 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:12.845199 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:12.845204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:12.845217 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:12.845471 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:12.845756 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:12.845759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:12.845760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:12.845908 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:12.846445 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:12.846450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:12.846651 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 14:50:12.846810 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 14:50:12.847022 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":126,"threads_started":1}
I20260926 14:50:12.848806 14779 heartbeater.cc:347] Connected to a master server at 127.12.68.254:41443
I20260926 14:50:12.848925 14779 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:12.849368 14779 heartbeater.cc:511] Master 127.12.68.254:41443 requested a full tablet report, sending...
I20260926 14:50:12.850313 14295 ts_manager.cc:194] Registered new tserver with Master: e4709deb2fde426091fdbda06ee64cfc (127.12.68.195:39993)
I20260926 14:50:12.850978 14295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:55097
I20260926 14:50:12.853681 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 14648
I20260926 14:50:12.853780 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:12.854053 12563 external_mini_cluster.cc:1428] Setting key 3ad4739596f0e7ce8ccb555097026e98
I20260926 14:50:12.855767 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:50:12.856961 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.850783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:43309 (local address 127.12.68.254:41443)
0926 14:50:12.850894 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:12.850897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:12.851027 (+ 130us) server_negotiation.cc:407] Connection header received
0926 14:50:12.851160 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:12.851162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:12.851204 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:12.851270 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:12.851676 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.852442 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.853183 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.853381 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.853476 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.854385 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:12.854388 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:12.854390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:12.854409 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:12.855576 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.855953 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.855957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.855958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.856028 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.856263 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.856266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.856268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.856421 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:12.856501 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:12.856737 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 14:50:12.856829 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:50:12.856971 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.850693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41443 (local address 127.12.68.195:43309)
0926 14:50:12.850952 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:12.850966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:12.851059 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:12.851292 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:12.851295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:12.851481 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:12.851575 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.851581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.852544 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.852547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.853079 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.853086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.853459 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.853461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.853520 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.853924 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:12.853934 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:12.854254 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:12.855677 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:12.855680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:12.855682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:12.855855 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:12.856105 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:12.856108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:12.856110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:12.856186 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:12.856504 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:12.856507 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:12.856587 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 14:50:12.856674 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 14:50:12.856793 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":68,"threads_started":1}
Sep 26 14:50:12 dist-test-slave-24mn krb5kdc[14251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:50:12 dist-test-slave-24mn krb5kdc[14251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:12.878497 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.871677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.12.68.254:41443)
0926 14:50:12.871780 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:12.871783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:12.871794 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:50:12.871832 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:12.871834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:12.871870 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:12.871969 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:12.872572 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.873053 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.873875 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.874075 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.874157 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.876146 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:12.876149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:12.876150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:12.876171 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:12.877344 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.877644 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.877647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.877648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.877711 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:12.877875 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:12.877877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:12.877878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:12.878011 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:12.878100 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:12.878278 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 14:50:12.878355 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:50:12.880895 14295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:12.883627 14295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:12.896677 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.891208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:39993 (local address 127.0.0.1:56674)
0926 14:50:12.891480 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:12.891523 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:12.891646 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:12.892698 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:12.892705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:12.892728 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:12.892899 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.892906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.894011 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.894015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.895333 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.895344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.896171 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.896174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.896276 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.896311 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 14:50:12.896412 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 14:50:12.896525 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":25,"thread_start_us":62,"threads_started":1}
I20260926 14:50:12.896840 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.891973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.12.68.193:44695)
0926 14:50:12.892361 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:12.892373 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:12.892393 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:50:12.892459 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:12.892464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:12.892662 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:12.892816 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:12.893032 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.894146 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.895482 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.896316 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.896422 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.896569 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:12.896637 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 14:50:12.896706 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":95,"threads_started":1}
I20260926 14:50:12.896835 14797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.891294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44695 (local address 127.0.0.1:41746)
0926 14:50:12.892006 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:12.892019 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:12.892098 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:12.892787 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:12.892789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:12.892802 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:12.892938 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.892944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.894256 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.894259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.895369 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.895376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.896429 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.896432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.896526 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.896546 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:50:12.896598 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 14:50:12.896647 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":28,"thread_start_us":84,"threads_started":1}
I20260926 14:50:12.897272 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.891875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:43401 (local address 127.0.0.1:47476)
0926 14:50:12.892134 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:12.892145 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:12.892208 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:12.893164 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:12.893167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:12.893183 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:12.893266 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.893272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.894484 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.894487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.895437 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.895447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.896728 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.896731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.897000 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.897022 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:50:12.897085 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:50:12.897160 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":88,"threads_started":1}
I20260926 14:50:12.897300 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.892381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.12.68.194:43401)
0926 14:50:12.892785 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:12.892796 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:12.892815 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:50:12.892877 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:12.892882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:12.893034 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:12.893189 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:12.893370 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.894384 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.895552 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.896330 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.896422 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.896568 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:12.897123 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 14:50:12.897200 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":98,"threads_started":1}
I20260926 14:50:12.898064 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.891320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.12.68.195:39993)
0926 14:50:12.892216 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:12.892229 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:12.892251 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:50:12.892320 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:12.892325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:12.892522 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:12.893030 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:12.893139 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.893912 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.895455 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.896060 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.896166 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.896323 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:12.897849 (+ 1526us) server_negotiation.cc:299] Negotiation successful
0926 14:50:12.897930 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":95,"threads_started":1}
I20260926 14:50:12.899791 14573 tablet_service.cc:1467] Processing CreateTablet for tablet 69fdca36f92a4146baee063d2649fffd (DEFAULT_TABLE table=test-table [id=27d87a5eb6b64c0a81cc6e03735d0a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:12.899776 14433 tablet_service.cc:1467] Processing CreateTablet for tablet 69fdca36f92a4146baee063d2649fffd (DEFAULT_TABLE table=test-table [id=27d87a5eb6b64c0a81cc6e03735d0a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:12.900295 14713 tablet_service.cc:1467] Processing CreateTablet for tablet 69fdca36f92a4146baee063d2649fffd (DEFAULT_TABLE table=test-table [id=27d87a5eb6b64c0a81cc6e03735d0a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:12.900714 14573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69fdca36f92a4146baee063d2649fffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:12.900746 14433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69fdca36f92a4146baee063d2649fffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:12.901134 14713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69fdca36f92a4146baee063d2649fffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:12.906745 14802 tablet_bootstrap.cc:492] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c: Bootstrap starting.
I20260926 14:50:12.908422 14803 tablet_bootstrap.cc:492] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7: Bootstrap starting.
I20260926 14:50:12.909051 14802 tablet_bootstrap.cc:654] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:12.909703 14802 log.cc:824] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:12.910024 14803 tablet_bootstrap.cc:654] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:12.910432 14804 tablet_bootstrap.cc:492] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc: Bootstrap starting.
I20260926 14:50:12.910770 14803 log.cc:824] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:12.911798 14802 tablet_bootstrap.cc:492] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c: No bootstrap required, opened a new log
I20260926 14:50:12.912009 14802 ts_tablet_manager.cc:1397] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:50:12.912351 14804 tablet_bootstrap.cc:654] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:12.912647 14803 tablet_bootstrap.cc:492] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7: No bootstrap required, opened a new log
I20260926 14:50:12.912814 14803 ts_tablet_manager.cc:1397] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:50:12.913187 14804 log.cc:824] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:12.914776 14804 tablet_bootstrap.cc:492] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc: No bootstrap required, opened a new log
I20260926 14:50:12.914932 14804 ts_tablet_manager.cc:1397] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:50:12.915696 14802 raft_consensus.cc:348] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } }
I20260926 14:50:12.915967 14802 raft_consensus.cc:374] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:12.916036 14802 raft_consensus.cc:729] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7430a561f644e57888513349abf504c, State: Initialized, Role: FOLLOWER
I20260926 14:50:12.916371 14803 raft_consensus.cc:348] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } }
I20260926 14:50:12.916487 14802 consensus_queue.cc:260] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } }
I20260926 14:50:12.916610 14803 raft_consensus.cc:374] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:12.916664 14803 raft_consensus.cc:729] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a259b4b95e4261b7e86713bbf992d7, State: Initialized, Role: FOLLOWER
I20260926 14:50:12.917089 14639 heartbeater.cc:503] Master 127.12.68.254:41443 was elected leader, sending a full tablet report...
I20260926 14:50:12.917086 14803 consensus_queue.cc:260] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } }
I20260926 14:50:12.917443 14802 ts_tablet_manager.cc:1428] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:50:12.917898 14803 ts_tablet_manager.cc:1428] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:50:12.918184 14499 heartbeater.cc:503] Master 127.12.68.254:41443 was elected leader, sending a full tablet report...
I20260926 14:50:12.918262 14804 raft_consensus.cc:348] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } }
I20260926 14:50:12.918476 14804 raft_consensus.cc:374] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:12.918519 14804 raft_consensus.cc:729] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4709deb2fde426091fdbda06ee64cfc, State: Initialized, Role: FOLLOWER
I20260926 14:50:12.918884 14804 consensus_queue.cc:260] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } }
I20260926 14:50:12.919749 14804 ts_tablet_manager.cc:1428] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:50:12.920272 14779 heartbeater.cc:503] Master 127.12.68.254:41443 was elected leader, sending a full tablet report...
W20260926 14:50:12.943778 14500 tablet.cc:2369] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:12.964083 14810 raft_consensus.cc:482] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:12.964223 14810 raft_consensus.cc:504] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } }
I20260926 14:50:12.965219 14810 leader_election.cc:288] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92a259b4b95e4261b7e86713bbf992d7 (127.12.68.193:44695), e7430a561f644e57888513349abf504c (127.12.68.194:43401)
I20260926 14:50:12.968989 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.965462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:59063 (local address 127.12.68.193:44695)
0926 14:50:12.965592 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:12.965596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:12.965661 (+ 65us) server_negotiation.cc:407] Connection header received
0926 14:50:12.965795 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:12.965798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:12.965844 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:12.965919 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:12.966154 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.966952 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.968101 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.968485 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.968586 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.968637 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:12.968799 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 14:50:12.968866 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:50:12.968999 14672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.965388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44695 (local address 127.12.68.195:59063)
0926 14:50:12.965587 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:12.965601 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:12.965706 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:12.965932 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:12.965933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:12.965946 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:12.966057 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.966063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.967052 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.967056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.967994 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.968003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.968583 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.968587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.968694 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.968716 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:50:12.968779 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:50:12.968847 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260926 14:50:12.969412 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.965428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:43401 (local address 127.12.68.195:56925)
0926 14:50:12.965734 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:12.965745 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:12.965821 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:12.966008 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:12.966010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:12.966023 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:12.966117 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.966123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.967090 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.967094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.968331 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:12.968340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.968939 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.968942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:12.969045 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.969194 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 14:50:12.969251 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:50:12.969311 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":83,"threads_started":1}
I20260926 14:50:12.969476 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:12.965549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:56925 (local address 127.12.68.194:43401)
0926 14:50:12.965668 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:12.965672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:12.965803 (+ 131us) server_negotiation.cc:407] Connection header received
0926 14:50:12.965892 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:12.965895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:12.965934 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:12.965996 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:12.966221 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.966952 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.968440 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:12.968842 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:12.968939 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:12.968994 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:12.969285 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 14:50:12.969360 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:50:12.969509 14453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69fdca36f92a4146baee063d2649fffd" candidate_uuid: "e4709deb2fde426091fdbda06ee64cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a259b4b95e4261b7e86713bbf992d7" is_pre_election: true
I20260926 14:50:12.969762 14453 raft_consensus.cc:2454] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4709deb2fde426091fdbda06ee64cfc in term 0.
I20260926 14:50:12.969952 14593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69fdca36f92a4146baee063d2649fffd" candidate_uuid: "e4709deb2fde426091fdbda06ee64cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7430a561f644e57888513349abf504c" is_pre_election: true
I20260926 14:50:12.970193 14593 raft_consensus.cc:2454] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4709deb2fde426091fdbda06ee64cfc in term 0.
I20260926 14:50:12.970197 14665 leader_election.cc:302] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92a259b4b95e4261b7e86713bbf992d7, e4709deb2fde426091fdbda06ee64cfc; no voters:
I20260926 14:50:12.970443 14810 raft_consensus.cc:2781] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:12.970538 14810 raft_consensus.cc:482] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:12.970589 14810 raft_consensus.cc:3037] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:12.971596 14810 raft_consensus.cc:504] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } }
I20260926 14:50:12.971912 14810 leader_election.cc:288] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [CANDIDATE]: Term 1 election: Requested vote from peers 92a259b4b95e4261b7e86713bbf992d7 (127.12.68.193:44695), e7430a561f644e57888513349abf504c (127.12.68.194:43401)
I20260926 14:50:12.972210 14593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69fdca36f92a4146baee063d2649fffd" candidate_uuid: "e4709deb2fde426091fdbda06ee64cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7430a561f644e57888513349abf504c"
I20260926 14:50:12.972224 14453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69fdca36f92a4146baee063d2649fffd" candidate_uuid: "e4709deb2fde426091fdbda06ee64cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a259b4b95e4261b7e86713bbf992d7"
I20260926 14:50:12.972314 14453 raft_consensus.cc:3037] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:12.972316 14593 raft_consensus.cc:3037] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:12.973237 14453 raft_consensus.cc:2454] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4709deb2fde426091fdbda06ee64cfc in term 1.
I20260926 14:50:12.973502 14665 leader_election.cc:302] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92a259b4b95e4261b7e86713bbf992d7, e4709deb2fde426091fdbda06ee64cfc; no voters:
I20260926 14:50:12.973639 14810 raft_consensus.cc:2781] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:12.973680 14593 raft_consensus.cc:2454] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4709deb2fde426091fdbda06ee64cfc in term 1.
I20260926 14:50:12.973809 14810 raft_consensus.cc:686] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [term 1 LEADER]: Becoming Leader. State: Replica: e4709deb2fde426091fdbda06ee64cfc, State: Running, Role: LEADER
I20260926 14:50:12.974087 14810 consensus_queue.cc:237] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } }
I20260926 14:50:12.976962 14296 catalog_manager.cc:5495] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc reported cstate change: term changed from 0 to 1, leader changed from <none> to e4709deb2fde426091fdbda06ee64cfc (127.12.68.195). New cstate: current_term: 1 leader_uuid: "e4709deb2fde426091fdbda06ee64cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } health_report { overall_health: UNKNOWN } } }
Sep 26 14:50:13 dist-test-slave-24mn krb5kdc[14251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:13.028916 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.021401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41443 (local address 127.0.0.1:45104)
0926 14:50:13.021708 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:13.021721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:13.021829 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:13.022031 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:13.022034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:13.022482 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:13.022573 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.022580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.023338 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.023342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.023908 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.023917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.024380 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.024383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.024453 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.024959 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:13.024971 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:13.026307 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:13.027644 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:13.027652 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:13.027657 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:13.027872 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:13.028107 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:13.028109 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:13.028110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:13.028187 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:13.028463 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:13.028468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:13.028570 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 14:50:13.028652 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 14:50:13.028760 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":111,"threads_started":1}
I20260926 14:50:13.028934 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.021548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.12.68.254:41443)
0926 14:50:13.021688 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:13.021692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:13.021787 (+ 95us) server_negotiation.cc:407] Connection header received
0926 14:50:13.021909 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:13.021912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:13.021945 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:13.022005 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:13.022695 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.023255 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.024014 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.024293 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.024377 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.026436 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:13.026439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:13.026441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:13.026462 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:13.027553 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:13.027972 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:13.027974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:13.027976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:13.028041 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:13.028247 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:13.028249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:13.028250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:13.028395 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:13.028460 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:13.028690 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 14:50:13.028814 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":55}
I20260926 14:50:13.032320 14296 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:45104:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 14:50:13.032933 14296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:13.037065 14433 tablet_service.cc:1467] Processing CreateTablet for tablet dd7f71eea5d9492aa4288a9bf6ae641e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=137449d9bbc648f8be66d3e08513ea40]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:13.037197 14713 tablet_service.cc:1467] Processing CreateTablet for tablet dd7f71eea5d9492aa4288a9bf6ae641e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=137449d9bbc648f8be66d3e08513ea40]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:13.037146 14573 tablet_service.cc:1467] Processing CreateTablet for tablet dd7f71eea5d9492aa4288a9bf6ae641e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=137449d9bbc648f8be66d3e08513ea40]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:13.037710 14433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7f71eea5d9492aa4288a9bf6ae641e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:13.037770 14573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7f71eea5d9492aa4288a9bf6ae641e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:13.037770 14713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7f71eea5d9492aa4288a9bf6ae641e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:13.040321 14802 tablet_bootstrap.cc:492] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c: Bootstrap starting.
I20260926 14:50:13.040321 14804 tablet_bootstrap.cc:492] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc: Bootstrap starting.
I20260926 14:50:13.040324 14803 tablet_bootstrap.cc:492] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7: Bootstrap starting.
I20260926 14:50:13.041047 14803 tablet_bootstrap.cc:654] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:13.041196 14804 tablet_bootstrap.cc:654] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:13.041196 14802 tablet_bootstrap.cc:654] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:13.042375 14804 tablet_bootstrap.cc:492] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc: No bootstrap required, opened a new log
I20260926 14:50:13.042455 14804 ts_tablet_manager.cc:1397] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:13.042564 14803 tablet_bootstrap.cc:492] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7: No bootstrap required, opened a new log
I20260926 14:50:13.042650 14803 ts_tablet_manager.cc:1397] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:13.042878 14804 raft_consensus.cc:348] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.042987 14804 raft_consensus.cc:374] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:13.043026 14804 raft_consensus.cc:729] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4709deb2fde426091fdbda06ee64cfc, State: Initialized, Role: FOLLOWER
I20260926 14:50:13.043037 14803 raft_consensus.cc:348] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.043133 14803 raft_consensus.cc:374] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:13.043170 14803 raft_consensus.cc:729] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a259b4b95e4261b7e86713bbf992d7, State: Initialized, Role: FOLLOWER
I20260926 14:50:13.043159 14804 consensus_queue.cc:260] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.043275 14803 consensus_queue.cc:260] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.043411 14804 ts_tablet_manager.cc:1428] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:13.043510 14803 ts_tablet_manager.cc:1428] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:50:13.043532 14809 tablet_replica.cc:439] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } }
I20260926 14:50:13.043587 14810 tablet_replica.cc:439] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } }
I20260926 14:50:13.043660 14809 tablet_replica.cc:442] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:13.043691 14810 tablet_replica.cc:442] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:13.044312 14802 tablet_bootstrap.cc:492] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c: No bootstrap required, opened a new log
I20260926 14:50:13.044384 14802 ts_tablet_manager.cc:1397] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 14:50:13.044752 14802 raft_consensus.cc:348] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.044855 14802 raft_consensus.cc:374] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:13.044888 14802 raft_consensus.cc:729] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7430a561f644e57888513349abf504c, State: Initialized, Role: FOLLOWER
I20260926 14:50:13.045006 14802 consensus_queue.cc:260] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.045217 14802 ts_tablet_manager.cc:1428] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:13.045396 14808 tablet_replica.cc:439] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } }
I20260926 14:50:13.045511 14808 tablet_replica.cc:442] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 14:50:13.096164 14780 tablet.cc:2369] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:50:13.139400 14640 tablet.cc:2369] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:13.146780 14809 raft_consensus.cc:482] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:13.146904 14809 raft_consensus.cc:504] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.147979 14809 leader_election.cc:288] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7430a561f644e57888513349abf504c (127.12.68.194:43401), e4709deb2fde426091fdbda06ee64cfc (127.12.68.195:39993)
I20260926 14:50:13.151724 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.148298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:51213 (local address 127.12.68.194:43401)
0926 14:50:13.148471 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:13.148475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:13.148690 (+ 215us) server_negotiation.cc:407] Connection header received
0926 14:50:13.148809 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:13.148812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:13.148848 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:13.148902 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:13.149189 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.149745 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.150772 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.151213 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.151322 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.151376 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:13.151541 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 14:50:13.151608 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 14:50:13.151758 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.148228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:43401 (local address 127.12.68.193:51213)
0926 14:50:13.148600 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:13.148620 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:13.148711 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:13.148916 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:13.148919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:13.148968 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:13.149090 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.149096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.149851 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.149855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.150661 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.150669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.151337 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.151341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.151434 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.151450 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 14:50:13.151513 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:50:13.151581 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":25,"spinlock_wait_cycles":768,"thread_start_us":76,"threads_started":1}
I20260926 14:50:13.151933 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.148309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:33103 (local address 127.12.68.195:39993)
0926 14:50:13.148455 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:13.148460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:13.148650 (+ 190us) server_negotiation.cc:407] Connection header received
0926 14:50:13.148774 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:13.148777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:13.148818 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:13.148878 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:13.149519 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.150008 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.151042 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.151433 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.151535 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.151583 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:13.151741 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 14:50:13.151806 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 14:50:13.151950 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.148226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:39993 (local address 127.12.68.193:33103)
0926 14:50:13.148569 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:13.148581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:13.148690 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:13.148895 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:13.148897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:13.148912 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:13.149419 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.149427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.150107 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.150111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.150918 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.150928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.151538 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.151541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.151642 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.151660 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:50:13.151720 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 14:50:13.151781 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"spinlock_wait_cycles":4608,"thread_start_us":108,"threads_started":1}
I20260926 14:50:13.152137 14593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7f71eea5d9492aa4288a9bf6ae641e" candidate_uuid: "92a259b4b95e4261b7e86713bbf992d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7430a561f644e57888513349abf504c" is_pre_election: true
I20260926 14:50:13.152288 14593 raft_consensus.cc:2454] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92a259b4b95e4261b7e86713bbf992d7 in term 0.
I20260926 14:50:13.152621 14386 leader_election.cc:302] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92a259b4b95e4261b7e86713bbf992d7, e7430a561f644e57888513349abf504c; no voters:
I20260926 14:50:13.152624 14733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7f71eea5d9492aa4288a9bf6ae641e" candidate_uuid: "92a259b4b95e4261b7e86713bbf992d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4709deb2fde426091fdbda06ee64cfc" is_pre_election: true
I20260926 14:50:13.152809 14733 raft_consensus.cc:2454] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92a259b4b95e4261b7e86713bbf992d7 in term 0.
I20260926 14:50:13.152863 14809 raft_consensus.cc:2781] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:13.152922 14809 raft_consensus.cc:482] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:13.152964 14809 raft_consensus.cc:3037] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:13.153944 14809 raft_consensus.cc:504] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.154248 14809 leader_election.cc:288] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [CANDIDATE]: Term 1 election: Requested vote from peers e7430a561f644e57888513349abf504c (127.12.68.194:43401), e4709deb2fde426091fdbda06ee64cfc (127.12.68.195:39993)
I20260926 14:50:13.154513 14593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7f71eea5d9492aa4288a9bf6ae641e" candidate_uuid: "92a259b4b95e4261b7e86713bbf992d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7430a561f644e57888513349abf504c"
I20260926 14:50:13.154548 14733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7f71eea5d9492aa4288a9bf6ae641e" candidate_uuid: "92a259b4b95e4261b7e86713bbf992d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4709deb2fde426091fdbda06ee64cfc"
I20260926 14:50:13.154637 14593 raft_consensus.cc:3037] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:13.154651 14733 raft_consensus.cc:3037] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:13.155580 14593 raft_consensus.cc:2454] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92a259b4b95e4261b7e86713bbf992d7 in term 1.
I20260926 14:50:13.155718 14733 raft_consensus.cc:2454] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92a259b4b95e4261b7e86713bbf992d7 in term 1.
I20260926 14:50:13.155800 14386 leader_election.cc:302] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92a259b4b95e4261b7e86713bbf992d7, e7430a561f644e57888513349abf504c; no voters:
I20260926 14:50:13.155994 14809 raft_consensus.cc:2781] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:13.156147 14809 raft_consensus.cc:686] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [term 1 LEADER]: Becoming Leader. State: Replica: 92a259b4b95e4261b7e86713bbf992d7, State: Running, Role: LEADER
I20260926 14:50:13.156438 14809 consensus_queue.cc:237] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.158370 14825 tablet_replica.cc:439] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7: TxnStatusTablet state changed. Reason: New leader 92a259b4b95e4261b7e86713bbf992d7. Latest consensus state: current_term: 1 leader_uuid: "92a259b4b95e4261b7e86713bbf992d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } }
I20260926 14:50:13.158466 14825 tablet_replica.cc:442] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:13.158710 14827 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 14:50:13.158934 14827 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 14:50:13.159315 14294 catalog_manager.cc:5495] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92a259b4b95e4261b7e86713bbf992d7 (127.12.68.193). New cstate: current_term: 1 leader_uuid: "92a259b4b95e4261b7e86713bbf992d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:13.193944 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.190102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44695 (local address 127.0.0.1:41762)
0926 14:50:13.190325 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:13.190337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:13.190426 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:13.190712 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:13.190714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:13.190720 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:13.190807 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.190811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.191768 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.191771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.192400 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.192409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.192916 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.192918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.192978 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.193042 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:13.193733 (+ 691us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:13.193740 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 14:50:13.193774 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 14:50:13.193836 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260926 14:50:13.193969 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.190209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.12.68.193:44695)
0926 14:50:13.190343 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:13.190347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:13.190379 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:50:13.190542 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:13.190545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:13.190605 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:13.190708 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:13.190895 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.191667 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.192492 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.192828 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.192925 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.193159 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:13.193660 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:13.193763 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:13.193794 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 14:50:13.193843 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:50:13.198772 14593 raft_consensus.cc:1261] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Refusing update from remote peer 92a259b4b95e4261b7e86713bbf992d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:13.198768 14733 raft_consensus.cc:1261] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Refusing update from remote peer 92a259b4b95e4261b7e86713bbf992d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:13.199375 14809 consensus_queue.cc:1035] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:13.199558 14825 consensus_queue.cc:1035] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:13.202351 14810 tablet_replica.cc:439] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc: TxnStatusTablet state changed. Reason: New leader 92a259b4b95e4261b7e86713bbf992d7. Latest consensus state: current_term: 1 leader_uuid: "92a259b4b95e4261b7e86713bbf992d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } }
I20260926 14:50:13.202467 14810 tablet_replica.cc:442] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:13.202775 14808 tablet_replica.cc:439] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c: TxnStatusTablet state changed. Reason: New leader 92a259b4b95e4261b7e86713bbf992d7. Latest consensus state: current_term: 1 leader_uuid: "92a259b4b95e4261b7e86713bbf992d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } }
I20260926 14:50:13.202872 14808 tablet_replica.cc:442] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:13.206571 14825 tablet_replica.cc:439] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92a259b4b95e4261b7e86713bbf992d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } }
I20260926 14:50:13.206678 14825 tablet_replica.cc:442] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:13.207268 14830 mvcc.cc:205] Tried to move back new op lower bound from 7333618537256095744 to 7333618537091031040. Current Snapshot: MvccSnapshot[applied={T|T < 7333618537256095744}]
I20260926 14:50:13.207715 14809 tablet_replica.cc:439] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92a259b4b95e4261b7e86713bbf992d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } }
I20260926 14:50:13.207810 14809 tablet_replica.cc:442] T dd7f71eea5d9492aa4288a9bf6ae641e P 92a259b4b95e4261b7e86713bbf992d7: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:13.208318 14808 tablet_replica.cc:439] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92a259b4b95e4261b7e86713bbf992d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } }
I20260926 14:50:13.208418 14808 tablet_replica.cc:442] T dd7f71eea5d9492aa4288a9bf6ae641e P e7430a561f644e57888513349abf504c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:13.208786 14810 tablet_replica.cc:439] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92a259b4b95e4261b7e86713bbf992d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } } }
I20260926 14:50:13.208884 14810 tablet_replica.cc:442] T dd7f71eea5d9492aa4288a9bf6ae641e P e4709deb2fde426091fdbda06ee64cfc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:13.209421 14832 mvcc.cc:205] Tried to move back new op lower bound from 7333618537256095744 to 7333618537091031040. Current Snapshot: MvccSnapshot[applied={T|T < 7333618537256095744}]
I20260926 14:50:13.210382 14831 mvcc.cc:205] Tried to move back new op lower bound from 7333618537256095744 to 7333618537091031040. Current Snapshot: MvccSnapshot[applied={T|T < 7333618537256095744}]
I20260926 14:50:13.220225 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.216784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.12.68.195:39993)
0926 14:50:13.216884 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:13.216889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:13.216902 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:13.216963 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:13.216965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:13.217029 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:13.217119 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:13.217356 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.217832 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.218980 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.219193 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.219264 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.219510 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:13.219863 (+ 353us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:13.219985 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:13.220077 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 14:50:13.220121 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:50:13.229868 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.226952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44695 (local address 127.12.68.195:51161)
0926 14:50:13.227090 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:13.227104 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:13.227180 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:13.227384 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:13.227387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:13.227393 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:13.227488 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.227493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.228224 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.228227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.228799 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.228807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.229217 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.229221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.229296 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.229334 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:13.229617 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:13.229622 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:13.229678 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:50:13.229749 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 14:50:13.229868 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.227071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:51161 (local address 127.12.68.193:44695)
0926 14:50:13.227192 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:13.227196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:13.227219 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:50:13.227264 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:13.227266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:13.227306 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:13.227362 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:13.227578 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.228129 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.228900 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.229121 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.229204 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.229467 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:13.229541 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:13.229596 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:13.229694 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 14:50:13.229741 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:50:13.236627 14453 raft_consensus.cc:1261] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Refusing update from remote peer e4709deb2fde426091fdbda06ee64cfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:13.237035 14593 raft_consensus.cc:1261] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Refusing update from remote peer e4709deb2fde426091fdbda06ee64cfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:13.237128 14810 consensus_queue.cc:1035] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:13.237407 14812 consensus_queue.cc:1035] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:13.259138 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.253956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:37277 (local address 127.12.68.195:39993)
0926 14:50:13.254058 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:13.254063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:13.256201 (+ 2138us) server_negotiation.cc:407] Connection header received
0926 14:50:13.256336 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:13.256339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:13.256387 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:13.256458 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:13.256693 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.257354 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.258214 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.258427 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.258513 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.258756 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:13.258823 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:13.258876 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:13.258969 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 14:50:13.259015 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:50:13.259156 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.253841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:39993 (local address 127.12.68.193:37277)
0926 14:50:13.256100 (+ 2259us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:13.256115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:13.256221 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:13.256481 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:13.256484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:13.256492 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:13.256588 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.256593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.257438 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.257442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.258101 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.258111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.258521 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.258525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.258604 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.258642 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:13.258897 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:13.258903 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:50:13.258967 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:50:13.259037 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2148,"thread_start_us":91,"threads_started":1}
I20260926 14:50:13.281416 14295 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:13.281597 14295 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:13.284293 14295 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f: Sending DeleteTablet for 3 replicas of tablet 69fdca36f92a4146baee063d2649fffd
I20260926 14:50:13.284991 14433 tablet_service.cc:1514] Processing DeleteTablet for tablet 69fdca36f92a4146baee063d2649fffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:13 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:41746
I20260926 14:50:13.285105 14713 tablet_service.cc:1514] Processing DeleteTablet for tablet 69fdca36f92a4146baee063d2649fffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:13 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:56674
I20260926 14:50:13.285434 14848 tablet_replica.cc:330] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7: stopping tablet replica
I20260926 14:50:13.285732 14848 raft_consensus.cc:2229] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:13.285882 14848 raft_consensus.cc:2258] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:13.285957 14573 tablet_service.cc:1514] Processing DeleteTablet for tablet 69fdca36f92a4146baee063d2649fffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:13 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:47476
I20260926 14:50:13.286507 14850 tablet_replica.cc:330] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c: stopping tablet replica
I20260926 14:50:13.286826 14850 raft_consensus.cc:2229] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:13.286847 14295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 14:50:13.287249 14850 raft_consensus.cc:2258] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:50:13.287264 14295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:13.288444 14849 tablet_replica.cc:330] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc: stopping tablet replica
I20260926 14:50:13.288712 14849 raft_consensus.cc:2229] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:13.289028 14849 raft_consensus.cc:2258] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:13.291606 14848 ts_tablet_manager.cc:1905] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:13.292457 14433 tablet_service.cc:1467] Processing CreateTablet for tablet eaba22a56a4e4e2c9509743f63fab2f9 (DEFAULT_TABLE table=test-table [id=0c8f3c32c70c4dcc8f1255592555918e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:13.292711 14433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaba22a56a4e4e2c9509743f63fab2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:13.295046 14848 ts_tablet_manager.cc:1918] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:13.295121 14848 log.cc:1196] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/wal/wals/69fdca36f92a4146baee063d2649fffd
I20260926 14:50:13.295280 14713 tablet_service.cc:1467] Processing CreateTablet for tablet eaba22a56a4e4e2c9509743f63fab2f9 (DEFAULT_TABLE table=test-table [id=0c8f3c32c70c4dcc8f1255592555918e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:13.295392 14848 ts_tablet_manager.cc:1939] T 69fdca36f92a4146baee063d2649fffd P 92a259b4b95e4261b7e86713bbf992d7: Deleting consensus metadata
I20260926 14:50:13.295521 14713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaba22a56a4e4e2c9509743f63fab2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:13.296785 14850 ts_tablet_manager.cc:1905] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:13.297909 14803 tablet_bootstrap.cc:492] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7: Bootstrap starting.
I20260926 14:50:13.298804 14803 tablet_bootstrap.cc:654] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:13.299342 14849 ts_tablet_manager.cc:1905] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:13.299394 14804 tablet_bootstrap.cc:492] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc: Bootstrap starting.
I20260926 14:50:13.299988 14281 catalog_manager.cc:4841] TS 92a259b4b95e4261b7e86713bbf992d7 (127.12.68.193:44695): tablet 69fdca36f92a4146baee063d2649fffd (table test-table [id=27d87a5eb6b64c0a81cc6e03735d0a8e]) successfully deleted
I20260926 14:50:13.301229 14804 tablet_bootstrap.cc:654] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:13.301831 14849 ts_tablet_manager.cc:1918] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:13.301910 14849 log.cc:1196] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/69fdca36f92a4146baee063d2649fffd
I20260926 14:50:13.302175 14849 ts_tablet_manager.cc:1939] T 69fdca36f92a4146baee063d2649fffd P e4709deb2fde426091fdbda06ee64cfc: Deleting consensus metadata
I20260926 14:50:13.302470 14573 tablet_service.cc:1467] Processing CreateTablet for tablet eaba22a56a4e4e2c9509743f63fab2f9 (DEFAULT_TABLE table=test-table [id=0c8f3c32c70c4dcc8f1255592555918e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:13.302682 14573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaba22a56a4e4e2c9509743f63fab2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:13.303110 14280 catalog_manager.cc:4841] TS e4709deb2fde426091fdbda06ee64cfc (127.12.68.195:39993): tablet 69fdca36f92a4146baee063d2649fffd (table test-table [id=27d87a5eb6b64c0a81cc6e03735d0a8e]) successfully deleted
I20260926 14:50:13.304533 14850 ts_tablet_manager.cc:1918] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:13.304608 14850 log.cc:1196] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/69fdca36f92a4146baee063d2649fffd
I20260926 14:50:13.304705 14802 tablet_bootstrap.cc:492] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c: Bootstrap starting.
I20260926 14:50:13.304859 14850 ts_tablet_manager.cc:1939] T 69fdca36f92a4146baee063d2649fffd P e7430a561f644e57888513349abf504c: Deleting consensus metadata
I20260926 14:50:13.305531 14802 tablet_bootstrap.cc:654] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:13.305846 14803 tablet_bootstrap.cc:492] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7: No bootstrap required, opened a new log
I20260926 14:50:13.305833 14282 catalog_manager.cc:4841] TS e7430a561f644e57888513349abf504c (127.12.68.194:43401): tablet 69fdca36f92a4146baee063d2649fffd (table test-table [id=27d87a5eb6b64c0a81cc6e03735d0a8e]) successfully deleted
I20260926 14:50:13.305917 14803 ts_tablet_manager.cc:1397] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260926 14:50:13.306311 14803 raft_consensus.cc:348] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.306413 14803 raft_consensus.cc:374] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:13.306439 14803 raft_consensus.cc:729] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a259b4b95e4261b7e86713bbf992d7, State: Initialized, Role: FOLLOWER
I20260926 14:50:13.306547 14803 consensus_queue.cc:260] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.306586 14804 tablet_bootstrap.cc:492] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc: No bootstrap required, opened a new log
I20260926 14:50:13.306651 14804 ts_tablet_manager.cc:1397] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260926 14:50:13.306762 14803 ts_tablet_manager.cc:1428] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:13.307015 14804 raft_consensus.cc:348] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.307117 14804 raft_consensus.cc:374] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:13.307147 14804 raft_consensus.cc:729] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4709deb2fde426091fdbda06ee64cfc, State: Initialized, Role: FOLLOWER
I20260926 14:50:13.307243 14804 consensus_queue.cc:260] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.307471 14804 ts_tablet_manager.cc:1428] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:13.309231 14802 tablet_bootstrap.cc:492] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c: No bootstrap required, opened a new log
I20260926 14:50:13.309314 14802 ts_tablet_manager.cc:1397] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 14:50:13.309721 14802 raft_consensus.cc:348] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.309834 14802 raft_consensus.cc:374] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:13.309876 14802 raft_consensus.cc:729] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7430a561f644e57888513349abf504c, State: Initialized, Role: FOLLOWER
I20260926 14:50:13.309988 14802 consensus_queue.cc:260] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.310231 14802 ts_tablet_manager.cc:1428] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:13.477118 14808 raft_consensus.cc:482] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:13.477232 14808 raft_consensus.cc:504] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.478261 14808 leader_election.cc:288] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92a259b4b95e4261b7e86713bbf992d7 (127.12.68.193:44695), e4709deb2fde426091fdbda06ee64cfc (127.12.68.195:39993)
I20260926 14:50:13.481786 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.478585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:60809 (local address 127.12.68.195:39993)
0926 14:50:13.478706 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:13.478710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:13.478899 (+ 189us) server_negotiation.cc:407] Connection header received
0926 14:50:13.479002 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:13.479004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:13.479040 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:13.479110 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:13.479379 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.479842 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.480896 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.481264 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.481355 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.481407 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:13.481607 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 14:50:13.481669 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:50:13.481788 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.478553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:39443 (local address 127.12.68.193:44695)
0926 14:50:13.478679 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:13.478683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:13.478809 (+ 126us) server_negotiation.cc:407] Connection header received
0926 14:50:13.478950 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:13.478953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:13.479003 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:13.479068 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:13.479339 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.479840 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.480889 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.481272 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.481376 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.481426 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:13.481614 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 14:50:13.481678 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:50:13.481834 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.478448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44695 (local address 127.12.68.194:39443)
0926 14:50:13.478717 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:13.478730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:13.478839 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:13.479091 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:13.479094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:13.479116 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:13.479247 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.479252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.479944 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.479948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.480775 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.480783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.481380 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.481383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.481495 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.481525 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 14:50:13.481590 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:50:13.481655 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":37,"spinlock_wait_cycles":9984,"thread_start_us":62,"threads_started":1}
I20260926 14:50:13.482002 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.478515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:39993 (local address 127.12.68.194:60809)
0926 14:50:13.478823 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:13.478834 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:13.478926 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:13.479125 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:13.479128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:13.479167 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:13.479284 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.479289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.479953 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.479956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.480785 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.480793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.481363 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.481367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.481487 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.481513 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:50:13.481569 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:50:13.481631 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":69,"spinlock_wait_cycles":2816,"thread_start_us":95,"threads_started":1}
I20260926 14:50:13.482183 14453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaba22a56a4e4e2c9509743f63fab2f9" candidate_uuid: "e7430a561f644e57888513349abf504c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a259b4b95e4261b7e86713bbf992d7" is_pre_election: true
I20260926 14:50:13.482293 14453 raft_consensus.cc:2454] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7430a561f644e57888513349abf504c in term 0.
I20260926 14:50:13.482589 14525 leader_election.cc:302] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92a259b4b95e4261b7e86713bbf992d7, e7430a561f644e57888513349abf504c; no voters:
I20260926 14:50:13.482712 14733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaba22a56a4e4e2c9509743f63fab2f9" candidate_uuid: "e7430a561f644e57888513349abf504c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4709deb2fde426091fdbda06ee64cfc" is_pre_election: true
I20260926 14:50:13.482833 14733 raft_consensus.cc:2454] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7430a561f644e57888513349abf504c in term 0.
I20260926 14:50:13.482832 14808 raft_consensus.cc:2781] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:13.482887 14808 raft_consensus.cc:482] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:13.482930 14808 raft_consensus.cc:3037] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:13.483832 14808 raft_consensus.cc:504] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.484150 14808 leader_election.cc:288] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [CANDIDATE]: Term 1 election: Requested vote from peers 92a259b4b95e4261b7e86713bbf992d7 (127.12.68.193:44695), e4709deb2fde426091fdbda06ee64cfc (127.12.68.195:39993)
I20260926 14:50:13.484447 14733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaba22a56a4e4e2c9509743f63fab2f9" candidate_uuid: "e7430a561f644e57888513349abf504c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4709deb2fde426091fdbda06ee64cfc"
I20260926 14:50:13.484447 14453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaba22a56a4e4e2c9509743f63fab2f9" candidate_uuid: "e7430a561f644e57888513349abf504c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a259b4b95e4261b7e86713bbf992d7"
I20260926 14:50:13.484555 14453 raft_consensus.cc:3037] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:13.484555 14733 raft_consensus.cc:3037] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:13.485566 14453 raft_consensus.cc:2454] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7430a561f644e57888513349abf504c in term 1.
I20260926 14:50:13.485566 14733 raft_consensus.cc:2454] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7430a561f644e57888513349abf504c in term 1.
I20260926 14:50:13.485821 14524 leader_election.cc:302] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 92a259b4b95e4261b7e86713bbf992d7, e4709deb2fde426091fdbda06ee64cfc, e7430a561f644e57888513349abf504c; no voters:
I20260926 14:50:13.486019 14808 raft_consensus.cc:2781] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:13.486207 14808 raft_consensus.cc:686] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [term 1 LEADER]: Becoming Leader. State: Replica: e7430a561f644e57888513349abf504c, State: Running, Role: LEADER
I20260926 14:50:13.486421 14808 consensus_queue.cc:237] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:13.488453 14295 catalog_manager.cc:5495] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c reported cstate change: term changed from 0 to 1, leader changed from <none> to e7430a561f644e57888513349abf504c (127.12.68.194). New cstate: current_term: 1 leader_uuid: "e7430a561f644e57888513349abf504c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:13.501511 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.498479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.12.68.194:43401)
0926 14:50:13.498594 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:13.498597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:13.498609 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:13.498738 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:13.498740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:13.498787 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:13.498896 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:13.499066 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.499533 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.500365 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.500567 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.500640 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.500808 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:13.501219 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:13.501297 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:13.501351 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 14:50:13.501392 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:50:13.510252 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.507656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:51461 (local address 127.12.68.193:44695)
0926 14:50:13.507754 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:13.507757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:13.507915 (+ 158us) server_negotiation.cc:407] Connection header received
0926 14:50:13.508012 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:13.508014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:13.508051 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:13.508115 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:13.508301 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.508766 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.509488 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.509691 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.509755 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.509958 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:13.510013 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:13.510064 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:13.510121 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 14:50:13.510160 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 14:50:13.510298 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.507577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44695 (local address 127.12.68.194:51461)
0926 14:50:13.507818 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:13.507831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:13.507937 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:13.508132 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:13.508133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:13.508138 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:13.508233 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.508237 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.508834 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.508836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.509412 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.509418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.509776 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.509778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.509836 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.509872 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:13.510068 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:13.510071 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:13.510109 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 14:50:13.510190 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260926 14:50:13.515137 14733 raft_consensus.cc:1261] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Refusing update from remote peer e7430a561f644e57888513349abf504c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:13.515252 14453 raft_consensus.cc:1261] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Refusing update from remote peer e7430a561f644e57888513349abf504c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:13.515576 14856 consensus_queue.cc:1035] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:13.515738 14808 consensus_queue.cc:1035] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [LEADER]: Connected to new peer: Peer: permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:13.531936 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.528979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:43401 (local address 127.12.68.193:54447)
0926 14:50:13.529143 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:13.529156 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:13.529216 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:13.529514 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:13.529517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:13.529524 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:13.529615 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.529621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.530341 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.530343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.530893 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:13.530901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.531318 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.531322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:13.531400 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.531430 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:13.531680 (+ 250us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:13.531686 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:50:13.531741 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 14:50:13.531817 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 14:50:13.531942 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.529014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:54447 (local address 127.12.68.194:43401)
0926 14:50:13.529327 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:13.529331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:13.529342 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:50:13.529377 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:13.529379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:13.529419 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:13.529489 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:13.529717 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.530254 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.531009 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:13.531216 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:13.531318 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:13.531536 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:13.531609 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:13.531679 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:13.531749 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 14:50:13.531797 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251}
W20260926 14:50:13.532665 14832 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 14:50:15.084579 14294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:15.084710 14294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:15.086750 14294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f: Sending DeleteTablet for 3 replicas of tablet eaba22a56a4e4e2c9509743f63fab2f9
I20260926 14:50:15.087190 14573 tablet_service.cc:1514] Processing DeleteTablet for tablet eaba22a56a4e4e2c9509743f63fab2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:15 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:47476
I20260926 14:50:15.087172 14433 tablet_service.cc:1514] Processing DeleteTablet for tablet eaba22a56a4e4e2c9509743f63fab2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:15 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:41746
I20260926 14:50:15.087361 14713 tablet_service.cc:1514] Processing DeleteTablet for tablet eaba22a56a4e4e2c9509743f63fab2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:15 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:56674
I20260926 14:50:15.087474 14890 tablet_replica.cc:330] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c: stopping tablet replica
I20260926 14:50:15.087599 14890 raft_consensus.cc:2229] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:15.087643 14891 tablet_replica.cc:330] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc: stopping tablet replica
I20260926 14:50:15.087750 14891 raft_consensus.cc:2229] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:15.087849 14891 raft_consensus.cc:2258] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:15.087874 14890 raft_consensus.cc:2258] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:15.088093 14889 tablet_replica.cc:330] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7: stopping tablet replica
I20260926 14:50:15.088192 14889 raft_consensus.cc:2229] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:15.088270 14889 raft_consensus.cc:2258] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:15.088477 14890 ts_tablet_manager.cc:1905] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:15.088919 14891 ts_tablet_manager.cc:1905] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:15.089807 14889 ts_tablet_manager.cc:1905] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:15.090941 14890 ts_tablet_manager.cc:1918] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:50:15.091013 14890 log.cc:1196] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/eaba22a56a4e4e2c9509743f63fab2f9
I20260926 14:50:15.091266 14890 ts_tablet_manager.cc:1939] T eaba22a56a4e4e2c9509743f63fab2f9 P e7430a561f644e57888513349abf504c: Deleting consensus metadata
I20260926 14:50:15.091836 14282 catalog_manager.cc:4841] TS e7430a561f644e57888513349abf504c (127.12.68.194:43401): tablet eaba22a56a4e4e2c9509743f63fab2f9 (table test-table [id=0c8f3c32c70c4dcc8f1255592555918e]) successfully deleted
I20260926 14:50:15.091985 14889 ts_tablet_manager.cc:1918] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:50:15.092054 14889 log.cc:1196] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/wal/wals/eaba22a56a4e4e2c9509743f63fab2f9
I20260926 14:50:15.092271 14889 ts_tablet_manager.cc:1939] T eaba22a56a4e4e2c9509743f63fab2f9 P 92a259b4b95e4261b7e86713bbf992d7: Deleting consensus metadata
I20260926 14:50:15.092803 14281 catalog_manager.cc:4841] TS 92a259b4b95e4261b7e86713bbf992d7 (127.12.68.193:44695): tablet eaba22a56a4e4e2c9509743f63fab2f9 (table test-table [id=0c8f3c32c70c4dcc8f1255592555918e]) successfully deleted
I20260926 14:50:15.093822 14891 ts_tablet_manager.cc:1918] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:50:15.093879 14891 log.cc:1196] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/eaba22a56a4e4e2c9509743f63fab2f9
I20260926 14:50:15.094076 14891 ts_tablet_manager.cc:1939] T eaba22a56a4e4e2c9509743f63fab2f9 P e4709deb2fde426091fdbda06ee64cfc: Deleting consensus metadata
I20260926 14:50:15.094594 14280 catalog_manager.cc:4841] TS e4709deb2fde426091fdbda06ee64cfc (127.12.68.195:39993): tablet eaba22a56a4e4e2c9509743f63fab2f9 (table test-table [id=0c8f3c32c70c4dcc8f1255592555918e]) successfully deleted
Sep 26 14:50:15 dist-test-slave-24mn krb5kdc[14251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:50:15 dist-test-slave-24mn krb5kdc[14251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:15.125556 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:15.104165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.12.68.254:41443)
0926 14:50:15.104379 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:15.104386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:15.104445 (+ 59us) server_negotiation.cc:407] Connection header received
0926 14:50:15.104545 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:15.104547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:15.104587 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:15.104662 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:15.105333 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:15.105874 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:15.106645 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:15.106866 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:15.106941 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:15.122802 (+ 15861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:15.122806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:15.122809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:15.122860 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:15.124285 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:15.124634 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:15.124636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:15.124638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:15.124710 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:15.124907 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:15.124909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:15.124910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:15.125079 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:15.125175 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:15.125330 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 14:50:15.125422 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":76,"threads_started":1}
I20260926 14:50:15.127722 14294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:15.128155 14294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:15.132194 14433 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6afb4ecdea4d229c24d7635e122cb2 (DEFAULT_TABLE table=test-table [id=fe0289cff02f454b80ae441a6d37c201]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:15.132215 14573 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6afb4ecdea4d229c24d7635e122cb2 (DEFAULT_TABLE table=test-table [id=fe0289cff02f454b80ae441a6d37c201]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:15.132455 14573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6afb4ecdea4d229c24d7635e122cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:15.132454 14433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6afb4ecdea4d229c24d7635e122cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:15.132432 14713 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6afb4ecdea4d229c24d7635e122cb2 (DEFAULT_TABLE table=test-table [id=fe0289cff02f454b80ae441a6d37c201]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:15.132645 14713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6afb4ecdea4d229c24d7635e122cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:15.134523 14900 tablet_bootstrap.cc:492] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc: Bootstrap starting.
I20260926 14:50:15.134788 14901 tablet_bootstrap.cc:492] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c: Bootstrap starting.
I20260926 14:50:15.135036 14902 tablet_bootstrap.cc:492] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7: Bootstrap starting.
I20260926 14:50:15.135653 14901 tablet_bootstrap.cc:654] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:15.135653 14900 tablet_bootstrap.cc:654] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:15.135772 14902 tablet_bootstrap.cc:654] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:15.136660 14900 tablet_bootstrap.cc:492] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc: No bootstrap required, opened a new log
I20260926 14:50:15.136662 14901 tablet_bootstrap.cc:492] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c: No bootstrap required, opened a new log
I20260926 14:50:15.136727 14901 ts_tablet_manager.cc:1397] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:15.136736 14900 ts_tablet_manager.cc:1397] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:50:15.137053 14900 raft_consensus.cc:348] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.137089 14901 raft_consensus.cc:348] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.137149 14900 raft_consensus.cc:374] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:15.137176 14901 raft_consensus.cc:374] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:15.137182 14900 raft_consensus.cc:729] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4709deb2fde426091fdbda06ee64cfc, State: Initialized, Role: FOLLOWER
I20260926 14:50:15.137195 14901 raft_consensus.cc:729] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7430a561f644e57888513349abf504c, State: Initialized, Role: FOLLOWER
I20260926 14:50:15.137292 14901 consensus_queue.cc:260] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.137295 14900 consensus_queue.cc:260] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.137527 14901 ts_tablet_manager.cc:1428] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:50:15.137615 14900 ts_tablet_manager.cc:1428] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:15.138355 14902 tablet_bootstrap.cc:492] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7: No bootstrap required, opened a new log
I20260926 14:50:15.138443 14902 ts_tablet_manager.cc:1397] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 14:50:15.138783 14902 raft_consensus.cc:348] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.138863 14902 raft_consensus.cc:374] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:15.138885 14902 raft_consensus.cc:729] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a259b4b95e4261b7e86713bbf992d7, State: Initialized, Role: FOLLOWER
I20260926 14:50:15.139029 14902 consensus_queue.cc:260] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.139285 14902 ts_tablet_manager.cc:1428] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:15.362155 14880 raft_consensus.cc:482] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:15.362282 14880 raft_consensus.cc:504] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.362703 14880 leader_election.cc:288] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92a259b4b95e4261b7e86713bbf992d7 (127.12.68.193:44695), e4709deb2fde426091fdbda06ee64cfc (127.12.68.195:39993)
I20260926 14:50:15.363152 14453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6afb4ecdea4d229c24d7635e122cb2" candidate_uuid: "e7430a561f644e57888513349abf504c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a259b4b95e4261b7e86713bbf992d7" is_pre_election: true
I20260926 14:50:15.363154 14732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6afb4ecdea4d229c24d7635e122cb2" candidate_uuid: "e7430a561f644e57888513349abf504c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4709deb2fde426091fdbda06ee64cfc" is_pre_election: true
I20260926 14:50:15.363308 14732 raft_consensus.cc:2454] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7430a561f644e57888513349abf504c in term 0.
I20260926 14:50:15.363308 14453 raft_consensus.cc:2454] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7430a561f644e57888513349abf504c in term 0.
I20260926 14:50:15.363631 14524 leader_election.cc:302] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4709deb2fde426091fdbda06ee64cfc, e7430a561f644e57888513349abf504c; no voters:
I20260926 14:50:15.363806 14880 raft_consensus.cc:2781] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:15.363864 14880 raft_consensus.cc:482] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:15.363968 14880 raft_consensus.cc:3037] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:15.365023 14880 raft_consensus.cc:504] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.365296 14880 leader_election.cc:288] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [CANDIDATE]: Term 1 election: Requested vote from peers 92a259b4b95e4261b7e86713bbf992d7 (127.12.68.193:44695), e4709deb2fde426091fdbda06ee64cfc (127.12.68.195:39993)
I20260926 14:50:15.365641 14732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6afb4ecdea4d229c24d7635e122cb2" candidate_uuid: "e7430a561f644e57888513349abf504c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4709deb2fde426091fdbda06ee64cfc"
I20260926 14:50:15.365641 14453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6afb4ecdea4d229c24d7635e122cb2" candidate_uuid: "e7430a561f644e57888513349abf504c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a259b4b95e4261b7e86713bbf992d7"
I20260926 14:50:15.365774 14453 raft_consensus.cc:3037] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:15.365774 14732 raft_consensus.cc:3037] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:15.366850 14732 raft_consensus.cc:2454] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7430a561f644e57888513349abf504c in term 1.
I20260926 14:50:15.366849 14453 raft_consensus.cc:2454] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7430a561f644e57888513349abf504c in term 1.
I20260926 14:50:15.367115 14524 leader_election.cc:302] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4709deb2fde426091fdbda06ee64cfc, e7430a561f644e57888513349abf504c; no voters:
I20260926 14:50:15.367277 14880 raft_consensus.cc:2781] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:15.367362 14880 raft_consensus.cc:686] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [term 1 LEADER]: Becoming Leader. State: Replica: e7430a561f644e57888513349abf504c, State: Running, Role: LEADER
I20260926 14:50:15.367519 14880 consensus_queue.cc:237] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.369035 14294 catalog_manager.cc:5495] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c reported cstate change: term changed from 0 to 1, leader changed from <none> to e7430a561f644e57888513349abf504c (127.12.68.194). New cstate: current_term: 1 leader_uuid: "e7430a561f644e57888513349abf504c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:15.432304 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:15.429364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.12.68.194:43401)
0926 14:50:15.429611 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:15.429617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:15.429627 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:50:15.429658 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:15.429660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:15.429698 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:15.429766 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:15.430014 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:15.430528 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:15.431329 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:15.431549 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:15.431631 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:15.431871 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:15.431967 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:15.432024 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:15.432119 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 14:50:15.432170 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":106,"threads_started":1}
I20260926 14:50:15.437317 14453 raft_consensus.cc:1261] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Refusing update from remote peer e7430a561f644e57888513349abf504c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:15.437316 14732 raft_consensus.cc:1261] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Refusing update from remote peer e7430a561f644e57888513349abf504c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:15.437662 14856 consensus_queue.cc:1035] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:15.437796 14880 consensus_queue.cc:1035] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [LEADER]: Connected to new peer: Peer: permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:15.469775 14295 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:15.469897 14295 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:15.471525 14295 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f: Sending DeleteTablet for 3 replicas of tablet 5a6afb4ecdea4d229c24d7635e122cb2
I20260926 14:50:15.472029 14573 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a6afb4ecdea4d229c24d7635e122cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:15 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:47476
I20260926 14:50:15.472040 14433 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a6afb4ecdea4d229c24d7635e122cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:15 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:41746
I20260926 14:50:15.472198 14713 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a6afb4ecdea4d229c24d7635e122cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:15 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:56674
I20260926 14:50:15.472245 14890 tablet_replica.cc:330] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c: stopping tablet replica
I20260926 14:50:15.472349 14889 tablet_replica.cc:330] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7: stopping tablet replica
I20260926 14:50:15.472353 14890 raft_consensus.cc:2229] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:15.472447 14889 raft_consensus.cc:2229] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:15.472514 14890 raft_consensus.cc:2258] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:15.472548 14889 raft_consensus.cc:2258] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:15.472689 14891 tablet_replica.cc:330] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc: stopping tablet replica
I20260926 14:50:15.472661 14295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 14:50:15.472795 14891 raft_consensus.cc:2229] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:15.472879 14891 raft_consensus.cc:2258] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:50:15.473054 14295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:15.473140 14890 ts_tablet_manager.cc:1905] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:15.473140 14889 ts_tablet_manager.cc:1905] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:15.473866 14891 ts_tablet_manager.cc:1905] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:15.475423 14889 ts_tablet_manager.cc:1918] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:15.475423 14890 ts_tablet_manager.cc:1918] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:15.475483 14889 log.cc:1196] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-0/wal/wals/5a6afb4ecdea4d229c24d7635e122cb2
I20260926 14:50:15.475499 14890 log.cc:1196] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/5a6afb4ecdea4d229c24d7635e122cb2
I20260926 14:50:15.475721 14890 ts_tablet_manager.cc:1939] T 5a6afb4ecdea4d229c24d7635e122cb2 P e7430a561f644e57888513349abf504c: Deleting consensus metadata
I20260926 14:50:15.475721 14889 ts_tablet_manager.cc:1939] T 5a6afb4ecdea4d229c24d7635e122cb2 P 92a259b4b95e4261b7e86713bbf992d7: Deleting consensus metadata
I20260926 14:50:15.476265 14891 ts_tablet_manager.cc:1918] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:15.476328 14891 log.cc:1196] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/5a6afb4ecdea4d229c24d7635e122cb2
I20260926 14:50:15.476310 14281 catalog_manager.cc:4841] TS 92a259b4b95e4261b7e86713bbf992d7 (127.12.68.193:44695): tablet 5a6afb4ecdea4d229c24d7635e122cb2 (table test-table [id=fe0289cff02f454b80ae441a6d37c201]) successfully deleted
I20260926 14:50:15.476323 14282 catalog_manager.cc:4841] TS e7430a561f644e57888513349abf504c (127.12.68.194:43401): tablet 5a6afb4ecdea4d229c24d7635e122cb2 (table test-table [id=fe0289cff02f454b80ae441a6d37c201]) successfully deleted
I20260926 14:50:15.476574 14891 ts_tablet_manager.cc:1939] T 5a6afb4ecdea4d229c24d7635e122cb2 P e4709deb2fde426091fdbda06ee64cfc: Deleting consensus metadata
I20260926 14:50:15.477027 14433 tablet_service.cc:1467] Processing CreateTablet for tablet 973c3d438a184e5ba6c8b6c8750ee07b (DEFAULT_TABLE table=test-table [id=f9aa310048094269b7c955a76c23f88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:15.477242 14433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973c3d438a184e5ba6c8b6c8750ee07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:15.477275 14713 tablet_service.cc:1467] Processing CreateTablet for tablet 973c3d438a184e5ba6c8b6c8750ee07b (DEFAULT_TABLE table=test-table [id=f9aa310048094269b7c955a76c23f88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:15.477468 14713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973c3d438a184e5ba6c8b6c8750ee07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:15.477813 14573 tablet_service.cc:1467] Processing CreateTablet for tablet 973c3d438a184e5ba6c8b6c8750ee07b (DEFAULT_TABLE table=test-table [id=f9aa310048094269b7c955a76c23f88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:15.478030 14573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973c3d438a184e5ba6c8b6c8750ee07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:15.479058 14900 tablet_bootstrap.cc:492] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc: Bootstrap starting.
I20260926 14:50:15.479177 14902 tablet_bootstrap.cc:492] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7: Bootstrap starting.
I20260926 14:50:15.479815 14900 tablet_bootstrap.cc:654] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:15.479907 14901 tablet_bootstrap.cc:492] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c: Bootstrap starting.
I20260926 14:50:15.479924 14280 catalog_manager.cc:4841] TS e4709deb2fde426091fdbda06ee64cfc (127.12.68.195:39993): tablet 5a6afb4ecdea4d229c24d7635e122cb2 (table test-table [id=fe0289cff02f454b80ae441a6d37c201]) successfully deleted
I20260926 14:50:15.479997 14902 tablet_bootstrap.cc:654] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:15.480621 14901 tablet_bootstrap.cc:654] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:15.481026 14902 tablet_bootstrap.cc:492] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7: No bootstrap required, opened a new log
I20260926 14:50:15.481108 14902 ts_tablet_manager.cc:1397] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:15.481271 14900 tablet_bootstrap.cc:492] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc: No bootstrap required, opened a new log
I20260926 14:50:15.481339 14900 ts_tablet_manager.cc:1397] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:50:15.481482 14902 raft_consensus.cc:348] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.481556 14901 tablet_bootstrap.cc:492] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c: No bootstrap required, opened a new log
I20260926 14:50:15.481580 14902 raft_consensus.cc:374] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:15.481613 14901 ts_tablet_manager.cc:1397] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:15.481611 14902 raft_consensus.cc:729] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a259b4b95e4261b7e86713bbf992d7, State: Initialized, Role: FOLLOWER
I20260926 14:50:15.481654 14900 raft_consensus.cc:348] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.481736 14900 raft_consensus.cc:374] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:15.481716 14902 consensus_queue.cc:260] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.481765 14900 raft_consensus.cc:729] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4709deb2fde426091fdbda06ee64cfc, State: Initialized, Role: FOLLOWER
I20260926 14:50:15.481858 14900 consensus_queue.cc:260] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.481951 14902 ts_tablet_manager.cc:1428] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:15.481931 14901 raft_consensus.cc:348] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.482031 14901 raft_consensus.cc:374] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:15.482060 14901 raft_consensus.cc:729] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7430a561f644e57888513349abf504c, State: Initialized, Role: FOLLOWER
I20260926 14:50:15.482091 14900 ts_tablet_manager.cc:1428] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:15.482162 14901 consensus_queue.cc:260] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.482390 14901 ts_tablet_manager.cc:1428] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:15.845865 14867 raft_consensus.cc:482] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:15.846011 14867 raft_consensus.cc:504] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.846393 14867 leader_election.cc:288] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7430a561f644e57888513349abf504c (127.12.68.194:43401), e4709deb2fde426091fdbda06ee64cfc (127.12.68.195:39993)
I20260926 14:50:15.846793 14593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973c3d438a184e5ba6c8b6c8750ee07b" candidate_uuid: "92a259b4b95e4261b7e86713bbf992d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7430a561f644e57888513349abf504c" is_pre_election: true
I20260926 14:50:15.846829 14732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973c3d438a184e5ba6c8b6c8750ee07b" candidate_uuid: "92a259b4b95e4261b7e86713bbf992d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4709deb2fde426091fdbda06ee64cfc" is_pre_election: true
I20260926 14:50:15.846967 14593 raft_consensus.cc:2454] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92a259b4b95e4261b7e86713bbf992d7 in term 0.
I20260926 14:50:15.846978 14732 raft_consensus.cc:2454] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92a259b4b95e4261b7e86713bbf992d7 in term 0.
I20260926 14:50:15.847225 14384 leader_election.cc:302] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92a259b4b95e4261b7e86713bbf992d7, e4709deb2fde426091fdbda06ee64cfc; no voters:
I20260926 14:50:15.847410 14867 raft_consensus.cc:2781] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:15.847484 14867 raft_consensus.cc:482] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:15.847518 14867 raft_consensus.cc:3037] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:15.848811 14867 raft_consensus.cc:504] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.849148 14867 leader_election.cc:288] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [CANDIDATE]: Term 1 election: Requested vote from peers e7430a561f644e57888513349abf504c (127.12.68.194:43401), e4709deb2fde426091fdbda06ee64cfc (127.12.68.195:39993)
I20260926 14:50:15.849457 14593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973c3d438a184e5ba6c8b6c8750ee07b" candidate_uuid: "92a259b4b95e4261b7e86713bbf992d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7430a561f644e57888513349abf504c"
I20260926 14:50:15.849474 14732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973c3d438a184e5ba6c8b6c8750ee07b" candidate_uuid: "92a259b4b95e4261b7e86713bbf992d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4709deb2fde426091fdbda06ee64cfc"
I20260926 14:50:15.849573 14593 raft_consensus.cc:3037] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:15.849572 14732 raft_consensus.cc:3037] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:15.850520 14732 raft_consensus.cc:2454] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92a259b4b95e4261b7e86713bbf992d7 in term 1.
I20260926 14:50:15.850585 14593 raft_consensus.cc:2454] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92a259b4b95e4261b7e86713bbf992d7 in term 1.
I20260926 14:50:15.850848 14384 leader_election.cc:302] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92a259b4b95e4261b7e86713bbf992d7, e4709deb2fde426091fdbda06ee64cfc; no voters:
I20260926 14:50:15.851018 14867 raft_consensus.cc:2781] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:15.851101 14867 raft_consensus.cc:686] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [term 1 LEADER]: Becoming Leader. State: Replica: 92a259b4b95e4261b7e86713bbf992d7, State: Running, Role: LEADER
I20260926 14:50:15.851228 14867 consensus_queue.cc:237] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } }
I20260926 14:50:15.852658 14295 catalog_manager.cc:5495] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92a259b4b95e4261b7e86713bbf992d7 (127.12.68.193). New cstate: current_term: 1 leader_uuid: "92a259b4b95e4261b7e86713bbf992d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a259b4b95e4261b7e86713bbf992d7" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:15.935046 14918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:15.932156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.12.68.193:44695)
0926 14:50:15.932385 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:15.932393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:15.932405 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:15.932443 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:15.932445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:15.932486 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:15.932553 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:15.932777 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:15.933298 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:15.934082 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:15.934311 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:15.934392 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:15.934627 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:15.934699 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:15.934753 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:15.934843 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 14:50:15.934895 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":78,"threads_started":1}
I20260926 14:50:15.942756 14918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:15.940268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:34751 (local address 127.12.68.193:44695)
0926 14:50:15.940360 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:15.940363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:15.940423 (+ 60us) server_negotiation.cc:407] Connection header received
0926 14:50:15.940523 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:15.940531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:15.940568 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:15.940616 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:15.940760 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:15.941209 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:15.941982 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:15.942201 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:15.942270 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:15.942439 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:15.942497 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:15.942551 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:15.942610 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 14:50:15.942647 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":35}
I20260926 14:50:15.942770 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:15.940132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44695 (local address 127.12.68.193:34751)
0926 14:50:15.940357 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:15.940368 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:15.940456 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:15.940627 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:15.940629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:15.940634 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:15.940707 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:15.940711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:15.941283 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:15.941285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:15.941898 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:15.941908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:15.942281 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:15.942283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:15.942337 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:15.942358 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:15.942555 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:15.942559 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:15.942602 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 14:50:15.942656 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260926 14:50:15.946898 14732 raft_consensus.cc:1261] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Refusing update from remote peer 92a259b4b95e4261b7e86713bbf992d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:15.947259 14593 raft_consensus.cc:1261] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Refusing update from remote peer 92a259b4b95e4261b7e86713bbf992d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:15.947225 14867 consensus_queue.cc:1035] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4709deb2fde426091fdbda06ee64cfc" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 39993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:15.947585 14867 consensus_queue.cc:1035] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7430a561f644e57888513349abf504c" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 14:50:15.958273 14868 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 14:50:17.506309 14294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:17.506457 14294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:17.508675 14294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f96d6cdeaaf4007a065b434aafa482f: Sending DeleteTablet for 3 replicas of tablet 973c3d438a184e5ba6c8b6c8750ee07b
I20260926 14:50:17.509107 14432 tablet_service.cc:1514] Processing DeleteTablet for tablet 973c3d438a184e5ba6c8b6c8750ee07b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:17 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:41746
I20260926 14:50:17.509198 14573 tablet_service.cc:1514] Processing DeleteTablet for tablet 973c3d438a184e5ba6c8b6c8750ee07b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:17 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:47476
I20260926 14:50:17.509296 14713 tablet_service.cc:1514] Processing DeleteTablet for tablet 973c3d438a184e5ba6c8b6c8750ee07b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:17 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:56674
I20260926 14:50:17.509531 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 14368
I20260926 14:50:17.509593 14945 tablet_replica.cc:330] T 973c3d438a184e5ba6c8b6c8750ee07b P 92a259b4b95e4261b7e86713bbf992d7: stopping tablet replica
I20260926 14:50:17.510465 14947 tablet_replica.cc:330] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc: stopping tablet replica
I20260926 14:50:17.510586 14947 raft_consensus.cc:2229] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:17.510676 14947 raft_consensus.cc:2258] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:17.509593 14946 tablet_replica.cc:330] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c: stopping tablet replica
I20260926 14:50:17.511265 14946 raft_consensus.cc:2229] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:17.511263 14947 ts_tablet_manager.cc:1905] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:17.511372 14946 raft_consensus.cc:2258] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:17.511937 14946 ts_tablet_manager.cc:1905] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:17.513902 14947 ts_tablet_manager.cc:1918] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:50:17.513980 14947 log.cc:1196] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/973c3d438a184e5ba6c8b6c8750ee07b
I20260926 14:50:17.514225 14947 ts_tablet_manager.cc:1939] T 973c3d438a184e5ba6c8b6c8750ee07b P e4709deb2fde426091fdbda06ee64cfc: Deleting consensus metadata
I20260926 14:50:17.514274 14946 ts_tablet_manager.cc:1918] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:50:17.514338 14946 log.cc:1196] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/973c3d438a184e5ba6c8b6c8750ee07b
I20260926 14:50:17.514552 14946 ts_tablet_manager.cc:1939] T 973c3d438a184e5ba6c8b6c8750ee07b P e7430a561f644e57888513349abf504c: Deleting consensus metadata
I20260926 14:50:17.514878 14280 catalog_manager.cc:4841] TS e4709deb2fde426091fdbda06ee64cfc (127.12.68.195:39993): tablet 973c3d438a184e5ba6c8b6c8750ee07b (table test-table [id=f9aa310048094269b7c955a76c23f88f]) successfully deleted
I20260926 14:50:17.515103 14282 catalog_manager.cc:4841] TS e7430a561f644e57888513349abf504c (127.12.68.194:43401): tablet 973c3d438a184e5ba6c8b6c8750ee07b (table test-table [id=f9aa310048094269b7c955a76c23f88f]) successfully deleted
W20260926 14:50:17.516211 14281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:44695 (error 108)
I20260926 14:50:17.516433 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 14508
I20260926 14:50:17.517577 14949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:17.517202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44695 (local address 127.0.0.1:41788)
0926 14:50:17.517403 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:17.517480 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:44695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
W20260926 14:50:17.517697 14281 catalog_manager.cc:4584] TS 92a259b4b95e4261b7e86713bbf992d7 (127.12.68.193:44695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 973c3d438a184e5ba6c8b6c8750ee07b: Network error: Client connection negotiation failed: client connection to 127.12.68.193:44695: connect: Connection refused (error 111)
I20260926 14:50:17.522686 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 14648
I20260926 14:50:17.527834 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 14264
2026-09-26T14:50:17Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6549 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:17 dist-test-slave-24mn krb5kdc[14952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:17 dist-test-slave-24mn krb5kdc[14952](info): set up 2 sockets
Sep 26 14:50:17 dist-test-slave-24mn krb5kdc[14952](info): commencing operation
krb5kdc: starting...
W20260926 14:50:18.479414 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:50:18 dist-test-slave-24mn krb5kdc[14952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:50:18.584645 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:33061
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:33061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:18.684593 14965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:18.684800 14965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:18.684835 14965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:18.688714 14965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:18.688761 14965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:18.688795 14965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:18.688812 14965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:18.688824 14965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:18.693674 14965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:33061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:33061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:18.694455 14965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:18.695277 14965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:18.703023 14971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:18.703130 14965 server_base.cc:1034] running on GCE node
W20260926 14:50:18.703106 14970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:18.703383 14973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:18.703873 14965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:18.704860 14965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:18.706032 14965 hybrid_clock.cc:648] HybridClock initialized: now 1790434218706009 us; error 34 us; skew 500 ppm
Sep 26 14:50:18 dist-test-slave-24mn krb5kdc[14952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434218, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:18.708457 14965 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:50:18.709261 14965 webserver.cc:460] Webserver started at http://127.12.68.254:33453/ using document root <none> and password file <none>
I20260926 14:50:18.709728 14965 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:18.709787 14965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:18.709960 14965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:18.711735 14965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "a12fdfab5c684bc4b34162aa9a2d3bfc"
format_stamp: "Formatted at 2026-09-26 14:50:18 on dist-test-slave-24mn"
server_key: "248fcc33f06b2a0513cffeb1a91df367"
server_key_iv: "7109029db3f695c28bdf2faa55ba0340"
server_key_version: "encryptionkey@0"
I20260926 14:50:18.712271 14965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "a12fdfab5c684bc4b34162aa9a2d3bfc"
format_stamp: "Formatted at 2026-09-26 14:50:18 on dist-test-slave-24mn"
server_key: "248fcc33f06b2a0513cffeb1a91df367"
server_key_iv: "7109029db3f695c28bdf2faa55ba0340"
server_key_version: "encryptionkey@0"
I20260926 14:50:18.715431 14965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:50:18.717501 14980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:18.718544 14965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:18.718643 14965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "a12fdfab5c684bc4b34162aa9a2d3bfc"
format_stamp: "Formatted at 2026-09-26 14:50:18 on dist-test-slave-24mn"
server_key: "248fcc33f06b2a0513cffeb1a91df367"
server_key_iv: "7109029db3f695c28bdf2faa55ba0340"
server_key_version: "encryptionkey@0"
I20260926 14:50:18.718724 14965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:18.734432 14965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:18.736791 14965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:18.736948 14965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:18.742769 14965 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:33061
I20260926 14:50:18.742835 15032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:33061 every 8 connection(s)
I20260926 14:50:18.743569 14965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:18.745962 15033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:18.749379 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 14965
I20260926 14:50:18.749477 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:18.749742 12563 external_mini_cluster.cc:1428] Setting key 0ea5e619da41002f39e5d49b8337d94d
I20260926 14:50:18.751260 15033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc: Bootstrap starting.
I20260926 14:50:18.753491 15033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:18.754130 15033 log.cc:824] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:18.755937 15033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc: No bootstrap required, opened a new log
Sep 26 14:50:18 dist-test-slave-24mn krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:18.759964 15033 raft_consensus.cc:348] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12fdfab5c684bc4b34162aa9a2d3bfc" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33061 } }
I20260926 14:50:18.760178 15033 raft_consensus.cc:374] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:18.760231 15033 raft_consensus.cc:729] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12fdfab5c684bc4b34162aa9a2d3bfc, State: Initialized, Role: FOLLOWER
I20260926 14:50:18.760689 15033 consensus_queue.cc:260] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12fdfab5c684bc4b34162aa9a2d3bfc" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33061 } }
I20260926 14:50:18.760881 15033 raft_consensus.cc:388] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:18.760950 15033 raft_consensus.cc:482] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:18.761050 15033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:18.761929 15033 raft_consensus.cc:504] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12fdfab5c684bc4b34162aa9a2d3bfc" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33061 } }
I20260926 14:50:18.762279 15033 leader_election.cc:302] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a12fdfab5c684bc4b34162aa9a2d3bfc; no voters:
I20260926 14:50:18.762547 15033 leader_election.cc:288] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:18.762660 15038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:18.762849 15038 raft_consensus.cc:686] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [term 1 LEADER]: Becoming Leader. State: Replica: a12fdfab5c684bc4b34162aa9a2d3bfc, State: Running, Role: LEADER
I20260926 14:50:18.763070 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:18.750888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.12.68.254:33061)
0926 14:50:18.751230 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:18.751244 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:18.751276 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:50:18.751869 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:18.751913 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:18.752215 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:18.752471 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:18.753272 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:18.754221 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:18.755242 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:18.755681 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:18.755787 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:18.758687 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:18.758692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:18.758703 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:18.758728 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:18.760301 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:18.760751 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:18.760755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:18.760759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:18.760877 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:18.761191 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:18.761194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:18.761195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:18.761547 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:18.761695 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:18.761970 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 14:50:18.762214 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":86,"threads_started":1}
I20260926 14:50:18.763180 15038 consensus_queue.cc:237] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12fdfab5c684bc4b34162aa9a2d3bfc" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33061 } }
I20260926 14:50:18.763680 15033 sys_catalog.cc:564] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:18.764289 15040 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a12fdfab5c684bc4b34162aa9a2d3bfc. Latest consensus state: current_term: 1 leader_uuid: "a12fdfab5c684bc4b34162aa9a2d3bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12fdfab5c684bc4b34162aa9a2d3bfc" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33061 } } }
I20260926 14:50:18.764479 15040 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:18.764675 15039 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a12fdfab5c684bc4b34162aa9a2d3bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12fdfab5c684bc4b34162aa9a2d3bfc" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33061 } } }
I20260926 14:50:18.764806 15039 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:18.765153 15047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:18.768050 15047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:18.773233 15047 catalog_manager.cc:1347] Generated new cluster ID: d8bb4f015a7e4642a319b313160e5890
I20260926 14:50:18.773280 15047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:18.783315 15047 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:18.784188 15047 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:18.791518 15047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a12fdfab5c684bc4b34162aa9a2d3bfc: Generated new TSK 0
I20260926 14:50:18.792114 15047 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:50:18.822295 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33061
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:18.957231 15059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:18.957468 15059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:18.957506 15059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:18.961344 15059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:18.961390 15059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:18.961441 15059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:18.966754 15059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:18.967655 15059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:18.968585 15059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:18.977058 15065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:18.977067 15064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:18.977239 15059 server_base.cc:1034] running on GCE node
W20260926 14:50:18.977059 15067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:18.977705 15059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:18.978226 15059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:18.979378 15059 hybrid_clock.cc:648] HybridClock initialized: now 1790434218979359 us; error 29 us; skew 500 ppm
Sep 26 14:50:18 dist-test-slave-24mn krb5kdc[14952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434218, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:18.981753 15059 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:50:18.982520 15059 webserver.cc:460] Webserver started at http://127.12.68.193:45719/ using document root <none> and password file <none>
I20260926 14:50:18.983003 15059 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:18.983058 15059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:18.983235 15059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:18.985177 15059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "ee1c92cdc3904ac78e47dd19d605898f"
format_stamp: "Formatted at 2026-09-26 14:50:18 on dist-test-slave-24mn"
server_key: "fbf296112a3f1715a0cb6435ea334abb"
server_key_iv: "40e87c4ec1aeb05e5ed743c4c7eb2aab"
server_key_version: "encryptionkey@0"
I20260926 14:50:18.985664 15059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "ee1c92cdc3904ac78e47dd19d605898f"
format_stamp: "Formatted at 2026-09-26 14:50:18 on dist-test-slave-24mn"
server_key: "fbf296112a3f1715a0cb6435ea334abb"
server_key_iv: "40e87c4ec1aeb05e5ed743c4c7eb2aab"
server_key_version: "encryptionkey@0"
I20260926 14:50:18.989256 15059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:18.991382 15074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:18.992494 15059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:18.992592 15059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "ee1c92cdc3904ac78e47dd19d605898f"
format_stamp: "Formatted at 2026-09-26 14:50:18 on dist-test-slave-24mn"
server_key: "fbf296112a3f1715a0cb6435ea334abb"
server_key_iv: "40e87c4ec1aeb05e5ed743c4c7eb2aab"
server_key_version: "encryptionkey@0"
I20260926 14:50:18.992674 15059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:19.011500 15059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:19.014240 15059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:19.014407 15059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:19.014933 15059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:19.015688 15059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:19.015733 15059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.015781 15059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:19.015808 15059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.023996 15059 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:40461
I20260926 14:50:19.024015 15187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:40461 every 8 connection(s)
I20260926 14:50:19.024797 15059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:19.028275 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 15059
I20260926 14:50:19.028385 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:19.028678 12563 external_mini_cluster.cc:1428] Setting key d1d8bc3b00153d3f8ae14e1fc0196091
Sep 26 14:50:19 dist-test-slave-24mn krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434218, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
I20260926 14:50:19.037302 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.026427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:43591 (local address 127.12.68.254:33061)
0926 14:50:19.026563 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:19.026567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:19.027530 (+ 963us) server_negotiation.cc:407] Connection header received
0926 14:50:19.028505 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:19.028509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:19.028556 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:19.028624 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:19.029794 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.030500 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.031620 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.031847 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.031968 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.034105 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:19.034108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:19.034110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:19.034133 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:19.035464 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.035939 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.035942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.035943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.036015 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.036301 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.036304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.036306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.036474 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:19.036555 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:19.037021 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 14:50:19.037144 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:50:19.038190 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.026776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:33061 (local address 127.12.68.193:43591)
0926 14:50:19.027399 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:19.027437 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:19.028284 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:19.028832 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:19.028841 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:19.029235 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:19.029690 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:19.029703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.030618 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.030620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:19.031511 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:19.031520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.031996 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.032000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:19.032154 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.032649 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:19.032663 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:19.033960 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:19.035576 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:19.035582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:19.035596 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:19.035825 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:19.036092 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:19.036095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:19.036097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:19.036220 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:19.036563 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:19.036569 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:19.036802 (+ 233us) client_negotiation.cc:769] Sending connection context
0926 14:50:19.036969 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 14:50:19.037263 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
I20260926 14:50:19.039315 15188 heartbeater.cc:347] Connected to a master server at 127.12.68.254:33061
I20260926 14:50:19.039526 15188 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:19.039992 15188 heartbeater.cc:511] Master 127.12.68.254:33061 requested a full tablet report, sending...
I20260926 14:50:19.041529 14997 ts_manager.cc:194] Registered new tserver with Master: ee1c92cdc3904ac78e47dd19d605898f (127.12.68.193:40461)
I20260926 14:50:19.042622 14997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:43591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:50:19.054461 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33061
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:19.158358 15193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:19.158608 15193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:19.158661 15193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:19.162534 15193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:19.162583 15193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:19.162657 15193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:19.167693 15193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:19.168628 15193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:19.169569 15193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:19.178159 15199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:19.178197 15201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:19.178197 15198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:19.178328 15193 server_base.cc:1034] running on GCE node
I20260926 14:50:19.178836 15193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:19.179385 15193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:19.180523 15193 hybrid_clock.cc:648] HybridClock initialized: now 1790434219180506 us; error 36 us; skew 500 ppm
Sep 26 14:50:19 dist-test-slave-24mn krb5kdc[14952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:19.182936 15193 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:50:19.183661 15193 webserver.cc:460] Webserver started at http://127.12.68.194:41983/ using document root <none> and password file <none>
I20260926 14:50:19.184170 15193 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:19.184226 15193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:19.184381 15193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:19.186337 15193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "ec70bc8bf3ef45e7868f9768e4194ab0"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-24mn"
server_key: "d17faa2c9d2807e359e93350e05be510"
server_key_iv: "8500b9e61aca233a8a7067a77f7c3bd7"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.186823 15193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "ec70bc8bf3ef45e7868f9768e4194ab0"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-24mn"
server_key: "d17faa2c9d2807e359e93350e05be510"
server_key_iv: "8500b9e61aca233a8a7067a77f7c3bd7"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.190202 15193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 14:50:19.192374 15208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.193456 15193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:19.193544 15193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "ec70bc8bf3ef45e7868f9768e4194ab0"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-24mn"
server_key: "d17faa2c9d2807e359e93350e05be510"
server_key_iv: "8500b9e61aca233a8a7067a77f7c3bd7"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.193636 15193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:19.208596 15193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:19.211248 15193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:19.211403 15193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:19.211971 15193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:19.212728 15193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:19.212771 15193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.212807 15193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:19.212831 15193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.220623 15193 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:35927
I20260926 14:50:19.220669 15321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:35927 every 8 connection(s)
I20260926 14:50:19.221442 15193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:19.229781 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 15193
I20260926 14:50:19.229902 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:19.230175 12563 external_mini_cluster.cc:1428] Setting key fb558006b7022dc973c3197aca71cf3a
Sep 26 14:50:19 dist-test-slave-24mn krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:19.234036 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.223146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:39837 (local address 127.12.68.254:33061)
0926 14:50:19.223295 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:19.223300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:19.224142 (+ 842us) server_negotiation.cc:407] Connection header received
0926 14:50:19.225322 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:19.225325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:19.225372 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:19.225446 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:19.226517 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.227045 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.228130 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.228361 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.228437 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.230718 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:19.230721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:19.230723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:19.230745 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:19.232178 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.232600 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.232603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.232605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.232683 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.232952 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.232955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.232957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.233169 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:19.233241 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:19.233801 (+ 560us) server_negotiation.cc:299] Negotiation successful
0926 14:50:19.233913 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:50:19.234818 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.223537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:33061 (local address 127.12.68.194:39837)
0926 14:50:19.224006 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:19.224045 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:19.225132 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:19.225591 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:19.225598 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:19.225935 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:19.226378 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:19.226390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.227136 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.227139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:19.228033 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:19.228041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.228469 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.228471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:19.228599 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.229121 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:19.229136 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:19.230571 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:19.232275 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:19.232280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:19.232291 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:19.232520 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:19.232754 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:19.232756 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:19.232758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:19.232871 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:19.233254 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:19.233261 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:19.233540 (+ 279us) client_negotiation.cc:769] Sending connection context
0926 14:50:19.233730 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 14:50:19.234062 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":116,"threads_started":1}
I20260926 14:50:19.235774 15322 heartbeater.cc:347] Connected to a master server at 127.12.68.254:33061
I20260926 14:50:19.236007 15322 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:19.236441 15322 heartbeater.cc:511] Master 127.12.68.254:33061 requested a full tablet report, sending...
I20260926 14:50:19.237317 14997 ts_manager.cc:194] Registered new tserver with Master: ec70bc8bf3ef45e7868f9768e4194ab0 (127.12.68.194:35927)
I20260926 14:50:19.237844 14997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:39837
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:50:19.254101 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33061
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:19.355010 15327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:19.355222 15327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:19.355261 15327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:19.359076 15327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:19.359120 15327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:19.359179 15327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:19.364022 15327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:19.364874 15327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:19.365756 15327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:19.374191 15333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:19.374249 15332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:19.374362 15327 server_base.cc:1034] running on GCE node
W20260926 14:50:19.374269 15335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:19.375176 15327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:19.375707 15327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:19.376856 15327 hybrid_clock.cc:648] HybridClock initialized: now 1790434219376828 us; error 39 us; skew 500 ppm
Sep 26 14:50:19 dist-test-slave-24mn krb5kdc[14952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:19.379467 15327 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:50:19.380241 15327 webserver.cc:460] Webserver started at http://127.12.68.195:46131/ using document root <none> and password file <none>
I20260926 14:50:19.380708 15327 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:19.380764 15327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:19.380947 15327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:19.382730 15327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "d494a515fd03470f870bb11bb8ff05e7"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-24mn"
server_key: "95297d60986c36f1743af7e88cf2da05"
server_key_iv: "7f7b65e28fa6d0647cf18ab784d5c0c2"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.383234 15327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "d494a515fd03470f870bb11bb8ff05e7"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-24mn"
server_key: "95297d60986c36f1743af7e88cf2da05"
server_key_iv: "7f7b65e28fa6d0647cf18ab784d5c0c2"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.386566 15327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:50:19.389148 15342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.390470 15327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:19.390560 15327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "d494a515fd03470f870bb11bb8ff05e7"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-24mn"
server_key: "95297d60986c36f1743af7e88cf2da05"
server_key_iv: "7f7b65e28fa6d0647cf18ab784d5c0c2"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.390654 15327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:19.406514 15327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:19.409264 15327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:19.409408 15327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:19.409922 15327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:19.410633 15327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:19.410676 15327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.410713 15327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:19.410741 15327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.418522 15327 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:43197
I20260926 14:50:19.418597 15455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:43197 every 8 connection(s)
I20260926 14:50:19.419309 15327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
Sep 26 14:50:19 dist-test-slave-24mn krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:19.428917 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 15327
I20260926 14:50:19.429018 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:19.429246 12563 external_mini_cluster.cc:1428] Setting key bf03574ab2461cdb5e10ddc2a6d8f02f
I20260926 14:50:19.431258 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.420898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:52201 (local address 127.12.68.254:33061)
0926 14:50:19.421070 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:19.421075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:19.421964 (+ 889us) server_negotiation.cc:407] Connection header received
0926 14:50:19.422836 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:19.422839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:19.422885 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:19.422946 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:19.424004 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.424525 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.425568 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.425791 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.425869 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.427966 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:19.427969 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:19.427971 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:19.427994 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:19.429348 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.429866 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.429869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.429871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.429965 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.430302 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.430305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.430307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.430516 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:19.430608 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:19.431033 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 14:50:19.431148 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 14:50:19.431999 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.421241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:33061 (local address 127.12.68.195:52201)
0926 14:50:19.421827 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:19.421863 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:19.422625 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:19.423088 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:19.423095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:19.423413 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:19.423838 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:19.423851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.424618 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.424621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:19.425461 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:19.425469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.425898 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.425900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:19.426024 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.426470 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:19.426485 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:19.427797 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:19.429462 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:19.429467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:19.429479 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:19.429760 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:19.430050 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:19.430053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:19.430054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:19.430215 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:19.430609 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:19.430614 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:19.430809 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 14:50:19.430977 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 14:50:19.431247 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":101,"threads_started":1}
I20260926 14:50:19.432873 15456 heartbeater.cc:347] Connected to a master server at 127.12.68.254:33061
I20260926 14:50:19.433048 15456 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:19.433427 15456 heartbeater.cc:511] Master 127.12.68.254:33061 requested a full tablet report, sending...
I20260926 14:50:19.434242 14997 ts_manager.cc:194] Registered new tserver with Master: d494a515fd03470f870bb11bb8ff05e7 (127.12.68.195:43197)
I20260926 14:50:19.434751 14997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:52201
I20260926 14:50:19.442451 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 14:50:19.453222 15036 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:19.452091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.12.68.254:33061)
0926 14:50:19.452202 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:19.452206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:19.452381 (+ 175us) server_negotiation.cc:407] Connection header received
0926 14:50:19.452497 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:19.452499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:19.452535 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:19.452589 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:19.453131 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60696: BlockingRecv error: recv got EOF from 127.0.0.1:60696 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:50:19.454228 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 15059
I20260926 14:50:19.458729 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 15193
I20260926 14:50:19.462780 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 15327
I20260926 14:50:19.466984 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 14965
2026-09-26T14:50:19Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1938 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:19 dist-test-slave-24mn krb5kdc[15468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:19 dist-test-slave-24mn krb5kdc[15468](info): set up 2 sockets
Sep 26 14:50:19 dist-test-slave-24mn krb5kdc[15468](info): commencing operation
krb5kdc: starting...
W20260926 14:50:20.415593 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:50:20 dist-test-slave-24mn krb5kdc[15468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:50:20.524986 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:39901
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:39901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:20.631626 15481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:20.631875 15481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:20.631958 15481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:20.635897 15481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:20.635950 15481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:20.635977 15481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:20.635998 15481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:20.636023 15481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:20.641104 15481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:39901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:39901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:20.642081 15481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:20.643052 15481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:20.651135 15489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:20.651181 15481 server_base.cc:1034] running on GCE node
W20260926 14:50:20.651135 15486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:20.651160 15487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:20.651922 15481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:20.652933 15481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:20.654096 15481 hybrid_clock.cc:648] HybridClock initialized: now 1790434220654074 us; error 34 us; skew 500 ppm
Sep 26 14:50:20 dist-test-slave-24mn krb5kdc[15468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434220, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:20.660157 15481 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:50:20.660948 15481 webserver.cc:460] Webserver started at http://127.12.68.254:36619/ using document root <none> and password file <none>
I20260926 14:50:20.661432 15481 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:20.661482 15481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:20.661643 15481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:20.663523 15481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "bd0c5cde700a40f3998e2d364d8ce603"
format_stamp: "Formatted at 2026-09-26 14:50:20 on dist-test-slave-24mn"
server_key: "ef67aac17da2aebe524fc4d507e8bf75"
server_key_iv: "f96f9a1adbee84350dfbd6a866241f96"
server_key_version: "encryptionkey@0"
I20260926 14:50:20.664062 15481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "bd0c5cde700a40f3998e2d364d8ce603"
format_stamp: "Formatted at 2026-09-26 14:50:20 on dist-test-slave-24mn"
server_key: "ef67aac17da2aebe524fc4d507e8bf75"
server_key_iv: "f96f9a1adbee84350dfbd6a866241f96"
server_key_version: "encryptionkey@0"
I20260926 14:50:20.667400 15481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:20.669612 15496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:20.670785 15481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:20.670892 15481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "bd0c5cde700a40f3998e2d364d8ce603"
format_stamp: "Formatted at 2026-09-26 14:50:20 on dist-test-slave-24mn"
server_key: "ef67aac17da2aebe524fc4d507e8bf75"
server_key_iv: "f96f9a1adbee84350dfbd6a866241f96"
server_key_version: "encryptionkey@0"
I20260926 14:50:20.670984 15481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:20.687568 15481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:20.690399 15481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:20.690562 15481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:20.696730 15481 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:39901
I20260926 14:50:20.696977 15548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:39901 every 8 connection(s)
I20260926 14:50:20.697599 15481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:20.700235 15549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:20.700311 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 15481
I20260926 14:50:20.700416 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:20.700677 12563 external_mini_cluster.cc:1428] Setting key c54d80eb578884947865eeff2dc2955f
I20260926 14:50:20.706468 15549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603: Bootstrap starting.
I20260926 14:50:20.708967 15549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603: Neither blocks nor log segments found. Creating new log.
Sep 26 14:50:20 dist-test-slave-24mn krb5kdc[15468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:20.709637 15549 log.cc:824] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:20.711690 15549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603: No bootstrap required, opened a new log
I20260926 14:50:20.714690 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:20.701825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.12.68.254:39901)
0926 14:50:20.702192 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:20.702205 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:20.702239 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:50:20.703085 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:20.703101 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:20.703403 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:20.703669 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:20.704548 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:20.705470 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:20.706588 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:20.707045 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:20.707151 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:20.709666 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:20.709671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:20.709683 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:20.709707 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:20.711514 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:20.712012 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:20.712018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:20.712025 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:20.712168 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:20.712478 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:20.712481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:20.712482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:20.712955 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:20.713138 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:20.713328 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 14:50:20.713636 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":105,"threads_started":1}
I20260926 14:50:20.716131 15549 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0c5cde700a40f3998e2d364d8ce603" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 39901 } }
I20260926 14:50:20.716338 15549 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:20.716393 15549 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0c5cde700a40f3998e2d364d8ce603, State: Initialized, Role: FOLLOWER
I20260926 14:50:20.716872 15549 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0c5cde700a40f3998e2d364d8ce603" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 39901 } }
I20260926 14:50:20.717005 15549 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:20.717067 15549 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:20.717152 15549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:20.717432 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:50:20.718199 15549 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0c5cde700a40f3998e2d364d8ce603" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 39901 } }
I20260926 14:50:20.718551 15549 leader_election.cc:302] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd0c5cde700a40f3998e2d364d8ce603; no voters:
I20260926 14:50:20.718868 15549 leader_election.cc:288] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:20.718956 15554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:20.719159 15554 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [term 1 LEADER]: Becoming Leader. State: Replica: bd0c5cde700a40f3998e2d364d8ce603, State: Running, Role: LEADER
I20260926 14:50:20.719497 15554 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0c5cde700a40f3998e2d364d8ce603" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 39901 } }
I20260926 14:50:20.720291 15549 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:20.720992 15555 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd0c5cde700a40f3998e2d364d8ce603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0c5cde700a40f3998e2d364d8ce603" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 39901 } } }
I20260926 14:50:20.721102 15555 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:20.721460 15556 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd0c5cde700a40f3998e2d364d8ce603. Latest consensus state: current_term: 1 leader_uuid: "bd0c5cde700a40f3998e2d364d8ce603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0c5cde700a40f3998e2d364d8ce603" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 39901 } } }
I20260926 14:50:20.721603 15556 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:20.722496 15564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:20.725378 15564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:20.730744 15564 catalog_manager.cc:1347] Generated new cluster ID: 8e0eebbe3d0b49a494fb7a912ea50abe
I20260926 14:50:20.730795 15564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:20.737090 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:20.729643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40964 (local address 127.12.68.254:39901)
0926 14:50:20.729745 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:20.729750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:20.731136 (+ 1386us) server_negotiation.cc:407] Connection header received
0926 14:50:20.731252 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:20.731255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:20.731303 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:20.731387 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:20.731966 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:20.732710 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:20.733867 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:20.734213 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:20.734290 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:20.735142 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:20.735146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:20.735148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:20.735615 (+ 467us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:20.735646 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:20.735665 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:20.735739 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:20.736896 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0926 14:50:20.736956 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":36}
I20260926 14:50:20.749068 15564 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:20.750041 15564 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:20.763980 15564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd0c5cde700a40f3998e2d364d8ce603: Generated new TSK 0
I20260926 14:50:20.764616 15564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:50:20.773856 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 15481
2026-09-26T14:50:20Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1305 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:20 dist-test-slave-24mn krb5kdc[15582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:20 dist-test-slave-24mn krb5kdc[15582](info): set up 2 sockets
Sep 26 14:50:20 dist-test-slave-24mn krb5kdc[15582](info): commencing operation
krb5kdc: starting...
W20260926 14:50:21.719527 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:50:21 dist-test-slave-24mn krb5kdc[15582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:50:21.827095 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:42231
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:42231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:21.928871 15595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:21.929169 15595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:21.929278 15595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:21.934799 15595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:21.934852 15595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:21.934880 15595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:21.934907 15595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:21.934937 15595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:21.940451 15595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:42231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:42231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:21.941335 15595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:21.942240 15595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:21.950042 15600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:21.950033 15595 server_base.cc:1034] running on GCE node
W20260926 14:50:21.950063 15603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:21.950208 15601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:21.950886 15595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:21.951867 15595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:21.953069 15595 hybrid_clock.cc:648] HybridClock initialized: now 1790434221953057 us; error 24 us; skew 500 ppm
Sep 26 14:50:21 dist-test-slave-24mn krb5kdc[15582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434221, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:21.955551 15595 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:50:21.956349 15595 webserver.cc:460] Webserver started at http://127.12.68.254:35821/ using document root <none> and password file <none>
I20260926 14:50:21.956828 15595 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:21.956880 15595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:21.957034 15595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:21.958884 15595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "4772c75364d447e99472bcf0d9598439"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-24mn"
server_key: "537d48900003d24862902ff028693604"
server_key_iv: "6498ceae32b01197d41d8b5f4f140a70"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.959357 15595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "4772c75364d447e99472bcf0d9598439"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-24mn"
server_key: "537d48900003d24862902ff028693604"
server_key_iv: "6498ceae32b01197d41d8b5f4f140a70"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.962730 15595 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:50:21.964869 15610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:21.965951 15595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:21.966056 15595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "4772c75364d447e99472bcf0d9598439"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-24mn"
server_key: "537d48900003d24862902ff028693604"
server_key_iv: "6498ceae32b01197d41d8b5f4f140a70"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.966141 15595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:21.979403 15595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:21.981997 15595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:21.982151 15595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:21.988350 15595 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:42231
I20260926 14:50:21.988587 15662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:42231 every 8 connection(s)
I20260926 14:50:21.989212 15595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:21.991744 15663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:21.992061 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 15595
I20260926 14:50:21.992178 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:21.992468 12563 external_mini_cluster.cc:1428] Setting key 795762ba2a29f86248ba05da02431c2e
I20260926 14:50:21.998024 15663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439: Bootstrap starting.
I20260926 14:50:22.000450 15663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439: Neither blocks nor log segments found. Creating new log.
Sep 26 14:50:21 dist-test-slave-24mn krb5kdc[15582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:22.001109 15663 log.cc:824] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:22.002714 15663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439: No bootstrap required, opened a new log
I20260926 14:50:22.005178 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.993704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.12.68.254:42231)
0926 14:50:21.994078 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:21.994092 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:21.994131 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:50:21.994793 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:21.994810 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:21.995119 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:21.995374 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:21.996231 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.997155 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.998269 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.998706 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.998814 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:22.001284 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.001289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.001301 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:22.001326 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.002739 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:22.003126 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:22.003131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:22.003135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:22.003241 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:22.003481 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:22.003484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:22.003486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:22.003825 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.004033 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.004276 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.004529 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
I20260926 14:50:22.006194 15663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4772c75364d447e99472bcf0d9598439" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42231 } }
I20260926 14:50:22.006395 15663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:22.006449 15663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4772c75364d447e99472bcf0d9598439, State: Initialized, Role: FOLLOWER
I20260926 14:50:22.006853 15663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4772c75364d447e99472bcf0d9598439" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42231 } }
I20260926 14:50:22.006978 15663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:22.007071 15663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:22.007169 15663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:22.007980 15663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4772c75364d447e99472bcf0d9598439" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42231 } }
I20260926 14:50:22.008268 15663 leader_election.cc:302] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4772c75364d447e99472bcf0d9598439; no voters:
I20260926 14:50:22.008517 15663 leader_election.cc:288] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:22.008605 15668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:22.008813 15668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [term 1 LEADER]: Becoming Leader. State: Replica: 4772c75364d447e99472bcf0d9598439, State: Running, Role: LEADER
I20260926 14:50:22.009152 15668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4772c75364d447e99472bcf0d9598439" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42231 } }
I20260926 14:50:22.009585 15663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:22.010820 15670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4772c75364d447e99472bcf0d9598439. Latest consensus state: current_term: 1 leader_uuid: "4772c75364d447e99472bcf0d9598439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4772c75364d447e99472bcf0d9598439" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42231 } } }
I20260926 14:50:22.010809 15669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4772c75364d447e99472bcf0d9598439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4772c75364d447e99472bcf0d9598439" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42231 } } }
I20260926 14:50:22.010957 15669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:22.010957 15670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:22.011344 15679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:22.013964 15679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:22.018769 15679 catalog_manager.cc:1347] Generated new cluster ID: d7a6dee6f2b74ba0abe99df5f5fda34d
I20260926 14:50:22.018819 15679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:22.028189 15679 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:22.029007 15679 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:22.034199 15679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4772c75364d447e99472bcf0d9598439: Generated new TSK 0
I20260926 14:50:22.034727 15679 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:50:22.061863 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42231
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:22.165499 15689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:22.165726 15689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:22.165771 15689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:22.169751 15689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:22.169800 15689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:22.169853 15689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:22.174917 15689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:22.175832 15689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:22.176772 15689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:22.185491 15697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.185566 15695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.185566 15694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:22.185746 15689 server_base.cc:1034] running on GCE node
I20260926 14:50:22.186182 15689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:22.186743 15689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:22.187912 15689 hybrid_clock.cc:648] HybridClock initialized: now 1790434222187891 us; error 30 us; skew 500 ppm
Sep 26 14:50:22 dist-test-slave-24mn krb5kdc[15582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434222, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:22.190315 15689 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:50:22.191073 15689 webserver.cc:460] Webserver started at http://127.12.68.193:46109/ using document root <none> and password file <none>
I20260926 14:50:22.191566 15689 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:22.191629 15689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:22.191813 15689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:22.193755 15689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "f38dd4c7c7474c848b68516552ccacce"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-24mn"
server_key: "93ad873564ddd33684f5eb40611493b0"
server_key_iv: "949a9835072bd71ff7612eb464dacbe3"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.194257 15689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "f38dd4c7c7474c848b68516552ccacce"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-24mn"
server_key: "93ad873564ddd33684f5eb40611493b0"
server_key_iv: "949a9835072bd71ff7612eb464dacbe3"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.197715 15689 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:50:22.199954 15704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.201045 15689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:22.201155 15689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "f38dd4c7c7474c848b68516552ccacce"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-24mn"
server_key: "93ad873564ddd33684f5eb40611493b0"
server_key_iv: "949a9835072bd71ff7612eb464dacbe3"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.201244 15689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:22.225104 15689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:22.227783 15689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:22.227970 15689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:22.228503 15689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:22.229280 15689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:22.229326 15689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.229377 15689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:22.229406 15689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.237550 15689 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:37543
I20260926 14:50:22.237576 15817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:37543 every 8 connection(s)
I20260926 14:50:22.238348 15689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:50:22 dist-test-slave-24mn krb5kdc[15582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434222, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:22.246994 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 15689
I20260926 14:50:22.247109 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:22.247346 12563 external_mini_cluster.cc:1428] Setting key b987ad1f4ef7f91caedfc16a4b3eb99a
I20260926 14:50:22.251207 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.239962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:44339 (local address 127.12.68.254:42231)
0926 14:50:22.240104 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.240109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.240913 (+ 804us) server_negotiation.cc:407] Connection header received
0926 14:50:22.242129 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.242132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.242175 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.242242 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.243300 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.243825 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.244971 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.245198 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.245277 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:22.247499 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.247504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.247506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:22.247533 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.249417 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:22.249847 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:22.249850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:22.249852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:22.249934 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:22.250250 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:22.250253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:22.250255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:22.250427 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.250497 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.251005 (+ 508us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.251096 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:50:22.252184 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.240338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42231 (local address 127.12.68.193:44339)
0926 14:50:22.240773 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:22.240812 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:22.241937 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:22.242392 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:22.242399 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:22.242714 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:22.243147 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:22.243159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.243958 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.243962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:22.244863 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:22.244872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.245330 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.245333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:22.245479 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:22.245941 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:22.245956 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:22.247357 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:22.249529 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:22.249533 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:22.249544 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:22.249768 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:22.250004 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:22.250006 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:22.250008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:22.250162 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:22.250510 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:22.250516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:22.250787 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 14:50:22.250976 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 14:50:22.251274 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260926 14:50:22.253325 15818 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42231
I20260926 14:50:22.253522 15818 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:22.253932 15818 heartbeater.cc:511] Master 127.12.68.254:42231 requested a full tablet report, sending...
I20260926 14:50:22.255230 15627 ts_manager.cc:194] Registered new tserver with Master: f38dd4c7c7474c848b68516552ccacce (127.12.68.193:37543)
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
I20260926 14:50:22.256325 15627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:44339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:50:22.270798 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42231
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:22.372485 15823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:22.372713 15823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:22.372767 15823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:22.376685 15823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:22.376736 15823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:22.376812 15823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:22.381808 15823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:22.382685 15823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:22.383560 15823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:50:22.392158 15823 server_base.cc:1034] running on GCE node
W20260926 14:50:22.392120 15828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.392119 15831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.392340 15829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:22.392669 15823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:22.393210 15823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:22.394353 15823 hybrid_clock.cc:648] HybridClock initialized: now 1790434222394334 us; error 32 us; skew 500 ppm
Sep 26 14:50:22 dist-test-slave-24mn krb5kdc[15582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434222, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:22.396704 15823 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:50:22.397460 15823 webserver.cc:460] Webserver started at http://127.12.68.194:40965/ using document root <none> and password file <none>
I20260926 14:50:22.398011 15823 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:22.398090 15823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:22.398298 15823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:22.400635 15823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "a347854554c44d79bea76774ae86b8a8"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-24mn"
server_key: "4444a733981f84bb3c3ffddd4fa11b33"
server_key_iv: "d5963636d5ac39296e705e53e4a858ca"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.401095 15823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "a347854554c44d79bea76774ae86b8a8"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-24mn"
server_key: "4444a733981f84bb3c3ffddd4fa11b33"
server_key_iv: "d5963636d5ac39296e705e53e4a858ca"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.404332 15823 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:22.406450 15838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.407496 15823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:22.407583 15823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "a347854554c44d79bea76774ae86b8a8"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-24mn"
server_key: "4444a733981f84bb3c3ffddd4fa11b33"
server_key_iv: "d5963636d5ac39296e705e53e4a858ca"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.407675 15823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:22.418474 15823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:22.420733 15823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:22.420874 15823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:22.421346 15823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:22.422025 15823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:22.422067 15823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.422106 15823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:22.422133 15823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.429575 15823 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:46735
I20260926 14:50:22.429621 15951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:46735 every 8 connection(s)
I20260926 14:50:22.430317 15823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:22.435336 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 15823
I20260926 14:50:22.435464 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:22.435720 12563 external_mini_cluster.cc:1428] Setting key 6e6e8d19b235ae911615d7f7658b3119
Sep 26 14:50:22 dist-test-slave-24mn krb5kdc[15582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434222, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:22.442835 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.431855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:46595 (local address 127.12.68.254:42231)
0926 14:50:22.431988 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.431991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.432722 (+ 731us) server_negotiation.cc:407] Connection header received
0926 14:50:22.433951 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.433954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.434011 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.434078 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.435104 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.435834 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.437078 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.437377 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.437479 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:22.439784 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.439788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.439790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:22.439817 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.441086 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:22.441522 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:22.441525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:22.441527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:22.441601 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:22.441879 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:22.441882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:22.441884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:22.442053 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.442132 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.442570 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.442684 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 14:50:22.443562 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.432193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42231 (local address 127.12.68.194:46595)
0926 14:50:22.432595 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:22.432631 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:22.433761 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:22.434239 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:22.434246 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:22.434565 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:22.434976 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:22.434988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.435975 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.435979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:22.436966 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:22.436977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.437502 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.437505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:22.437635 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:22.438118 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:22.438133 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:22.439644 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:22.441184 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:22.441189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:22.441203 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:22.441438 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:22.441676 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:22.441679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:22.441681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:22.441802 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:22.442142 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:22.442148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:22.442359 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 14:50:22.442516 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 14:50:22.442791 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
I20260926 14:50:22.444545 15952 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42231
I20260926 14:50:22.444742 15952 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:22.445154 15952 heartbeater.cc:511] Master 127.12.68.254:42231 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
I20260926 14:50:22.446113 15627 ts_manager.cc:194] Registered new tserver with Master: a347854554c44d79bea76774ae86b8a8 (127.12.68.194:46735)
I20260926 14:50:22.446653 15627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:46595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:50:22.464522 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42231
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:22.567066 15957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:22.567287 15957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:22.567338 15957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:22.571261 15957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:22.571311 15957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:22.571385 15957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:22.576280 15957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:22.577158 15957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:22.578055 15957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:22.586745 15963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:22.586921 15957 server_base.cc:1034] running on GCE node
W20260926 14:50:22.586745 15962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.586740 15965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:22.587438 15957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:22.588044 15957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:22.589187 15957 hybrid_clock.cc:648] HybridClock initialized: now 1790434222589161 us; error 35 us; skew 500 ppm
Sep 26 14:50:22 dist-test-slave-24mn krb5kdc[15582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434222, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:22.591621 15957 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:50:22.592491 15957 webserver.cc:460] Webserver started at http://127.12.68.195:34617/ using document root <none> and password file <none>
I20260926 14:50:22.592990 15957 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:22.593050 15957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:22.593227 15957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:22.595152 15957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "2301f84365d24271ac9579c2451aa877"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-24mn"
server_key: "78157fdec70bd2138fffdbbbb581efa5"
server_key_iv: "5e87dc2953cfcb4787d96c42cd818353"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.595669 15957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "2301f84365d24271ac9579c2451aa877"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-24mn"
server_key: "78157fdec70bd2138fffdbbbb581efa5"
server_key_iv: "5e87dc2953cfcb4787d96c42cd818353"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.599068 15957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:50:22.601404 15972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.602662 15957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:22.602785 15957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "2301f84365d24271ac9579c2451aa877"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-24mn"
server_key: "78157fdec70bd2138fffdbbbb581efa5"
server_key_iv: "5e87dc2953cfcb4787d96c42cd818353"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.602873 15957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:22.614472 15957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:22.617233 15957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:22.617383 15957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:22.617889 15957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:22.618605 15957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:22.618649 15957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.618695 15957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:22.618728 15957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.626487 15957 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:35229
I20260926 14:50:22.626523 16085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:35229 every 8 connection(s)
I20260926 14:50:22.627260 15957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:22.629220 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 15957
I20260926 14:50:22.629328 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:22.629624 12563 external_mini_cluster.cc:1428] Setting key 523f55f4ed21f839a5d5f1919fabc58f
Sep 26 14:50:22 dist-test-slave-24mn krb5kdc[15582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434222, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:22.639287 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.628845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:56557 (local address 127.12.68.254:42231)
0926 14:50:22.628960 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.628964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.629824 (+ 860us) server_negotiation.cc:407] Connection header received
0926 14:50:22.631205 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.631209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.631264 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.631336 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.632537 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.633055 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.634115 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.634329 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.634410 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:22.636509 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.636512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.636515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:22.636540 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.637544 (+ 1004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:22.637972 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:22.637975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:22.637977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:22.638048 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:22.638334 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:22.638337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:22.638339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:22.638504 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.638582 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.639021 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.639134 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:50:22.640008 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.629201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42231 (local address 127.12.68.195:56557)
0926 14:50:22.629668 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:22.629720 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:22.631020 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:22.631602 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:22.631609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:22.631967 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:22.632409 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:22.632422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.633163 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.633165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:22.634006 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:22.634014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.634449 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.634453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:22.634599 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:22.635037 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:22.635051 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:22.636367 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:22.637639 (+ 1272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:22.637645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:22.637657 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:22.637884 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:22.638124 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:22.638127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:22.638129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:22.638253 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:22.638593 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:22.638599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:22.638816 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 14:50:22.638966 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 14:50:22.639260 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":141,"threads_started":1}
I20260926 14:50:22.640972 16086 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42231
I20260926 14:50:22.641157 16086 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:22.641579 16086 heartbeater.cc:511] Master 127.12.68.254:42231 requested a full tablet report, sending...
I20260926 14:50:22.642576 15627 ts_manager.cc:194] Registered new tserver with Master: 2301f84365d24271ac9579c2451aa877 (127.12.68.195:35229)
I20260926 14:50:22.643115 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:50:22.643311 15627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:56557
Sep 26 14:50:22 dist-test-slave-24mn krb5kdc[15582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434222, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 14:50:22 dist-test-slave-24mn krb5kdc[15582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434222, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:22.665701 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.658354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44690 (local address 127.12.68.254:42231)
0926 14:50:22.658482 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.658487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.658623 (+ 136us) server_negotiation.cc:407] Connection header received
0926 14:50:22.658744 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.658747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.658790 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.658864 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.659521 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.660079 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.660885 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.661093 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.661175 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:22.663267 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.663270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.663272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:22.663293 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.664419 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:22.664771 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:22.664773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:22.664774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:22.664842 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:22.665044 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:22.665046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:22.665047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:22.665232 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.665324 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.665471 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.665554 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 14:50:22.666074 15627 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44690
I20260926 14:50:22.666909 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 15689
I20260926 14:50:22.671794 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 15823
I20260926 14:50:22.676065 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 15957
I20260926 14:50:22.680207 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 15595
2026-09-26T14:50:22Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1907 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:22 dist-test-slave-24mn krb5kdc[16098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:22 dist-test-slave-24mn krb5kdc[16098](info): set up 2 sockets
Sep 26 14:50:22 dist-test-slave-24mn krb5kdc[16098](info): commencing operation
krb5kdc: starting...
W20260926 14:50:23.631649 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:50:23 dist-test-slave-24mn krb5kdc[16098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:50:23.750849 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:41899
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:41899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:23.854739 16111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:23.855018 16111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:23.855057 16111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:23.858980 16111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:23.859032 16111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:23.859050 16111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:23.859066 16111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:23.859079 16111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:23.864115 16111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:41899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:41899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:23.865043 16111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:23.865954 16111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:23.873790 16119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:23.873780 16117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:23.873764 16116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:23.874025 16111 server_base.cc:1034] running on GCE node
I20260926 14:50:23.874580 16111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:23.875591 16111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:23.876748 16111 hybrid_clock.cc:648] HybridClock initialized: now 1790434223876707 us; error 52 us; skew 500 ppm
Sep 26 14:50:23 dist-test-slave-24mn krb5kdc[16098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434223, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:23.879237 16111 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:50:23.880085 16111 webserver.cc:460] Webserver started at http://127.12.68.254:44797/ using document root <none> and password file <none>
I20260926 14:50:23.880580 16111 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:23.880640 16111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:23.880815 16111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:23.882720 16111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "29339ca47e014d1ca70615bedeb35411"
format_stamp: "Formatted at 2026-09-26 14:50:23 on dist-test-slave-24mn"
server_key: "5457ad1703ac9590f0e9a7030a78cb1e"
server_key_iv: "da11e5e7869b2a1eee851467fa8332d3"
server_key_version: "encryptionkey@0"
I20260926 14:50:23.883240 16111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "29339ca47e014d1ca70615bedeb35411"
format_stamp: "Formatted at 2026-09-26 14:50:23 on dist-test-slave-24mn"
server_key: "5457ad1703ac9590f0e9a7030a78cb1e"
server_key_iv: "da11e5e7869b2a1eee851467fa8332d3"
server_key_version: "encryptionkey@0"
I20260926 14:50:23.886667 16111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 14:50:23.888917 16126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:23.890048 16111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:23.890154 16111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "29339ca47e014d1ca70615bedeb35411"
format_stamp: "Formatted at 2026-09-26 14:50:23 on dist-test-slave-24mn"
server_key: "5457ad1703ac9590f0e9a7030a78cb1e"
server_key_iv: "da11e5e7869b2a1eee851467fa8332d3"
server_key_version: "encryptionkey@0"
I20260926 14:50:23.890251 16111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:23.925189 16111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:23.928211 16111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:23.928382 16111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:23.934688 16111 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:41899
I20260926 14:50:23.934707 16178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:41899 every 8 connection(s)
I20260926 14:50:23.935555 16111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:23.936039 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 16111
I20260926 14:50:23.936153 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:23.936455 12563 external_mini_cluster.cc:1428] Setting key 7e7d873d2986bfbadac38d292052e134
I20260926 14:50:23.938654 16179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:23.943934 16179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411: Bootstrap starting.
Sep 26 14:50:23 dist-test-slave-24mn krb5kdc[16098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:23.946121 16179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:23.946784 16179 log.cc:824] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:23.948555 16179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411: No bootstrap required, opened a new log
I20260926 14:50:23.949499 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:23.937802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57172 (local address 127.12.68.254:41899)
0926 14:50:23.938221 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:23.938235 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:23.938269 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:50:23.938913 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:23.938935 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:23.939264 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:23.939593 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:23.940363 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.941341 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.942139 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.942593 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.942729 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:23.945192 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:23.945197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:23.945210 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:23.945236 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:23.946717 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:23.947133 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:23.947137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:23.947141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:23.947244 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:23.947480 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:23.947483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:23.947485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:23.947856 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:23.948062 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:23.948309 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 14:50:23.948571 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":136,"threads_started":1}
I20260926 14:50:23.951788 16179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29339ca47e014d1ca70615bedeb35411" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41899 } }
I20260926 14:50:23.952031 16179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:23.952088 16179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29339ca47e014d1ca70615bedeb35411, State: Initialized, Role: FOLLOWER
I20260926 14:50:23.952469 16179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29339ca47e014d1ca70615bedeb35411" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41899 } }
I20260926 14:50:23.952590 16179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:23.952652 16179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:23.952745 16179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:23.953540 16179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29339ca47e014d1ca70615bedeb35411" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41899 } }
I20260926 14:50:23.953846 16179 leader_election.cc:302] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29339ca47e014d1ca70615bedeb35411; no voters:
I20260926 14:50:23.954107 16179 leader_election.cc:288] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:23.954203 16184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:23.954414 16184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [term 1 LEADER]: Becoming Leader. State: Replica: 29339ca47e014d1ca70615bedeb35411, State: Running, Role: LEADER
I20260926 14:50:23.954728 16184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29339ca47e014d1ca70615bedeb35411" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41899 } }
I20260926 14:50:23.955219 16179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:23.956230 16186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29339ca47e014d1ca70615bedeb35411. Latest consensus state: current_term: 1 leader_uuid: "29339ca47e014d1ca70615bedeb35411" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29339ca47e014d1ca70615bedeb35411" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41899 } } }
I20260926 14:50:23.956362 16186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:23.956511 16185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29339ca47e014d1ca70615bedeb35411" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29339ca47e014d1ca70615bedeb35411" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41899 } } }
I20260926 14:50:23.956677 16185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411 [sys.catalog]: This master's current role is: LEADER
W20260926 14:50:23.959692 16200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:50:23.959793 16200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:50:23.959877 16194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:23.960583 16194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:23.965231 16194 catalog_manager.cc:1347] Generated new cluster ID: 4f7cb336f1154e098da6ae4a558f19ec
I20260926 14:50:23.965289 16194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:23.976159 16194 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:23.977352 16194 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:23.984314 16194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29339ca47e014d1ca70615bedeb35411: Generated new TSK 0
I20260926 14:50:23.985038 16194 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:50:24.010905 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:41899
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:24.116276 16205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:24.116516 16205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:24.116559 16205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:24.120523 16205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:24.120577 16205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:24.120632 16205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:24.125837 16205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:41899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:24.126793 16205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:24.127753 16205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:24.136564 16210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:24.136559 16211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:24.136539 16213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:24.136857 16205 server_base.cc:1034] running on GCE node
I20260926 14:50:24.137210 16205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:24.137813 16205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:24.138969 16205 hybrid_clock.cc:648] HybridClock initialized: now 1790434224138952 us; error 32 us; skew 500 ppm
Sep 26 14:50:24 dist-test-slave-24mn krb5kdc[16098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434224, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:24.141506 16205 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:50:24.142258 16205 webserver.cc:460] Webserver started at http://127.12.68.193:40441/ using document root <none> and password file <none>
I20260926 14:50:24.142735 16205 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:24.142789 16205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:24.142947 16205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:24.144877 16205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "4c7b2f7e6f064299ac8395e0b60b15d7"
format_stamp: "Formatted at 2026-09-26 14:50:24 on dist-test-slave-24mn"
server_key: "c5c095f5c5a5ce47ac835020c041e2e5"
server_key_iv: "6279c340ad721311f6b3cc23c1166d2d"
server_key_version: "encryptionkey@0"
I20260926 14:50:24.145372 16205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "4c7b2f7e6f064299ac8395e0b60b15d7"
format_stamp: "Formatted at 2026-09-26 14:50:24 on dist-test-slave-24mn"
server_key: "c5c095f5c5a5ce47ac835020c041e2e5"
server_key_iv: "6279c340ad721311f6b3cc23c1166d2d"
server_key_version: "encryptionkey@0"
I20260926 14:50:24.148815 16205 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:50:24.151105 16220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:24.152318 16205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:50:24.152416 16205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "4c7b2f7e6f064299ac8395e0b60b15d7"
format_stamp: "Formatted at 2026-09-26 14:50:24 on dist-test-slave-24mn"
server_key: "c5c095f5c5a5ce47ac835020c041e2e5"
server_key_iv: "6279c340ad721311f6b3cc23c1166d2d"
server_key_version: "encryptionkey@0"
I20260926 14:50:24.152510 16205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:24.193110 16205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:24.196024 16205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:24.196172 16205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:24.196749 16205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:24.197504 16205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:24.197549 16205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:24.197587 16205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:24.197605 16205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:24.205586 16205 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:45507
I20260926 14:50:24.205618 16333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:45507 every 8 connection(s)
I20260926 14:50:24.206419 16205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:50:24 dist-test-slave-24mn krb5kdc[16098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434224, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:24.216653 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 16205
I20260926 14:50:24.216805 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:24.217108 12563 external_mini_cluster.cc:1428] Setting key efeabfdfef8fe46d86a97a0aea6bc8cf
I20260926 14:50:24.218921 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:24.208153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:37651 (local address 127.12.68.254:41899)
0926 14:50:24.208272 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:24.208276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:24.209100 (+ 824us) server_negotiation.cc:407] Connection header received
0926 14:50:24.210364 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:24.210367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:24.210412 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:24.210482 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:24.211554 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.212135 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.213194 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.213413 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.213480 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:24.215647 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:24.215650 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:24.215652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:24.215676 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:24.217102 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:24.217558 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:24.217561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:24.217562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:24.217635 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:24.217930 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:24.217933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:24.217935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:24.218108 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:24.218189 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:24.218643 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 14:50:24.218759 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:50:24.219687 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:24.208516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41899 (local address 127.12.68.193:37651)
0926 14:50:24.208954 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:24.208994 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:24.210171 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:24.210619 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:24.210626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:24.210953 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:24.211426 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:24.211439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.212246 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.212249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:24.213105 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:24.213113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.213517 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.213520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:24.213655 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:24.214176 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:24.214192 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:24.215509 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:24.217206 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:24.217212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:24.217224 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:24.217454 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:24.217715 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:24.217718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:24.217720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:24.217846 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:24.218200 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:24.218206 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:24.218427 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 14:50:24.218581 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 14:50:24.218866 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":125,"threads_started":1}
I20260926 14:50:24.220880 16334 heartbeater.cc:347] Connected to a master server at 127.12.68.254:41899
I20260926 14:50:24.221086 16334 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:24.221527 16334 heartbeater.cc:511] Master 127.12.68.254:41899 requested a full tablet report, sending...
I20260926 14:50:24.222901 16143 ts_manager.cc:194] Registered new tserver with Master: 4c7b2f7e6f064299ac8395e0b60b15d7 (127.12.68.193:45507)
I20260926 14:50:24.223987 16143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:37651
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:50:24.240549 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:41899
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:24.345326 16339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:24.345557 16339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:24.345597 16339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:24.349534 16339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:24.349583 16339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:24.349643 16339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:24.354749 16339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:41899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:24.355697 16339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:24.356726 16339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:24.365499 16347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:24.365521 16344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:24.365497 16345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:24.365679 16339 server_base.cc:1034] running on GCE node
I20260926 14:50:24.366183 16339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:24.366761 16339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:24.367925 16339 hybrid_clock.cc:648] HybridClock initialized: now 1790434224367904 us; error 29 us; skew 500 ppm
Sep 26 14:50:24 dist-test-slave-24mn krb5kdc[16098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434224, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:24.370384 16339 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:50:24.371150 16339 webserver.cc:460] Webserver started at http://127.12.68.194:36745/ using document root <none> and password file <none>
I20260926 14:50:24.371649 16339 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:24.371707 16339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:24.371906 16339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:24.373790 16339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "8ebc3c3989b04514922eddc835202fd6"
format_stamp: "Formatted at 2026-09-26 14:50:24 on dist-test-slave-24mn"
server_key: "d9f94763f833406f83b747c14c8774d7"
server_key_iv: "8879026b9a8f1e5fba73e06790f20226"
server_key_version: "encryptionkey@0"
I20260926 14:50:24.374292 16339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "8ebc3c3989b04514922eddc835202fd6"
format_stamp: "Formatted at 2026-09-26 14:50:24 on dist-test-slave-24mn"
server_key: "d9f94763f833406f83b747c14c8774d7"
server_key_iv: "8879026b9a8f1e5fba73e06790f20226"
server_key_version: "encryptionkey@0"
I20260926 14:50:24.377723 16339 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:50:24.379920 16354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:24.381068 16339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:24.381178 16339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "8ebc3c3989b04514922eddc835202fd6"
format_stamp: "Formatted at 2026-09-26 14:50:24 on dist-test-slave-24mn"
server_key: "d9f94763f833406f83b747c14c8774d7"
server_key_iv: "8879026b9a8f1e5fba73e06790f20226"
server_key_version: "encryptionkey@0"
I20260926 14:50:24.381273 16339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:24.411520 16339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:24.414323 16339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:24.414497 16339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:24.415037 16339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:24.415814 16339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:24.415863 16339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:24.415938 16339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:24.415964 16339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:24.424013 16339 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:35703
I20260926 14:50:24.424045 16467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:35703 every 8 connection(s)
I20260926 14:50:24.424827 16339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:24.426098 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 16339
I20260926 14:50:24.426195 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:24.426460 12563 external_mini_cluster.cc:1428] Setting key f3d36d49d2196a45a99d6deb66ad5efd
Sep 26 14:50:24 dist-test-slave-24mn krb5kdc[16098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434224, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
I20260926 14:50:24.438159 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:24.426541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:34791 (local address 127.12.68.254:41899)
0926 14:50:24.426678 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:24.426682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:24.427492 (+ 810us) server_negotiation.cc:407] Connection header received
0926 14:50:24.429131 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:24.429134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:24.429177 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:24.429244 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:24.430508 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.431025 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.432200 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.432429 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.432515 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:24.434860 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:24.434864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:24.434866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:24.434891 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:24.436308 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:24.436753 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:24.436757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:24.436758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:24.436850 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:24.437141 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:24.437144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:24.437145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:24.437335 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:24.437426 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:24.437892 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 14:50:24.438003 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 14:50:24.438874 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:24.426912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41899 (local address 127.12.68.194:34791)
0926 14:50:24.427363 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:24.427404 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:24.428922 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:24.429450 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:24.429458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:24.429881 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:24.430354 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:24.430366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.431134 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.431138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:24.432096 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:24.432105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.432555 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.432558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:24.432713 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:24.433199 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:24.433214 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:24.434708 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:24.436414 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:24.436420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:24.436433 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:24.436663 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:24.436930 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:24.436933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:24.436935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:24.437060 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:24.437438 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:24.437444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:24.437663 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 14:50:24.437821 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 14:50:24.438058 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":103,"threads_started":1}
I20260926 14:50:24.439973 16468 heartbeater.cc:347] Connected to a master server at 127.12.68.254:41899
I20260926 14:50:24.440204 16468 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:24.440714 16468 heartbeater.cc:511] Master 127.12.68.254:41899 requested a full tablet report, sending...
I20260926 14:50:24.441781 16143 ts_manager.cc:194] Registered new tserver with Master: 8ebc3c3989b04514922eddc835202fd6 (127.12.68.194:35703)
I20260926 14:50:24.442312 16143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:34791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:50:24.451923 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:41899
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:24.555815 16473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:24.556069 16473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:24.556113 16473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:24.560199 16473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:24.560248 16473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:24.560305 16473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:24.565526 16473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:41899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:24.566440 16473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:24.567364 16473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:24.576102 16478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:24.576150 16473 server_base.cc:1034] running on GCE node
W20260926 14:50:24.576083 16479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:24.576083 16481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:24.576699 16473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:24.577284 16473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:24.578442 16473 hybrid_clock.cc:648] HybridClock initialized: now 1790434224578425 us; error 30 us; skew 500 ppm
Sep 26 14:50:24 dist-test-slave-24mn krb5kdc[16098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434224, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:24.581014 16473 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:50:24.581799 16473 webserver.cc:460] Webserver started at http://127.12.68.195:44975/ using document root <none> and password file <none>
I20260926 14:50:24.582305 16473 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:24.582367 16473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:24.582541 16473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:24.584537 16473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "3cc4b27924354087a306ffea096bacc4"
format_stamp: "Formatted at 2026-09-26 14:50:24 on dist-test-slave-24mn"
server_key: "21ed1edd26689a927e8ab5fee1a56b5b"
server_key_iv: "ae4782d2c2aa611f71b9bdda9d1c16e5"
server_key_version: "encryptionkey@0"
I20260926 14:50:24.585067 16473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "3cc4b27924354087a306ffea096bacc4"
format_stamp: "Formatted at 2026-09-26 14:50:24 on dist-test-slave-24mn"
server_key: "21ed1edd26689a927e8ab5fee1a56b5b"
server_key_iv: "ae4782d2c2aa611f71b9bdda9d1c16e5"
server_key_version: "encryptionkey@0"
I20260926 14:50:24.588387 16473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:24.590559 16488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:24.591671 16473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:24.591770 16473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "3cc4b27924354087a306ffea096bacc4"
format_stamp: "Formatted at 2026-09-26 14:50:24 on dist-test-slave-24mn"
server_key: "21ed1edd26689a927e8ab5fee1a56b5b"
server_key_iv: "ae4782d2c2aa611f71b9bdda9d1c16e5"
server_key_version: "encryptionkey@0"
I20260926 14:50:24.591861 16473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:24.619423 16473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:24.622185 16473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:24.622357 16473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:24.622931 16473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:24.623719 16473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:24.623765 16473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:24.623805 16473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:24.623833 16473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:24.631724 16473 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:39217
I20260926 14:50:24.631754 16601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:39217 every 8 connection(s)
I20260926 14:50:24.632575 16473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:24.637467 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 16473
I20260926 14:50:24.637583 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:24.637833 12563 external_mini_cluster.cc:1428] Setting key 0bc734f70c42b0b854a09fd4cb8f4171
Sep 26 14:50:24 dist-test-slave-24mn krb5kdc[16098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434224, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:24.645249 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:24.634272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:38359 (local address 127.12.68.254:41899)
0926 14:50:24.634402 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:24.634405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:24.635186 (+ 781us) server_negotiation.cc:407] Connection header received
0926 14:50:24.636510 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:24.636514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:24.636571 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:24.636637 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:24.637840 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.638445 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.639940 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.640153 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.640240 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:24.642387 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:24.642390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:24.642392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:24.642424 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:24.643619 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:24.644111 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:24.644114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:24.644115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:24.644183 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:24.644445 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:24.644447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:24.644448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:24.644596 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:24.644661 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:24.645050 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 14:50:24.645137 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:50:24.645999 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:24.634609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41899 (local address 127.12.68.195:38359)
0926 14:50:24.635039 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:24.635078 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:24.636305 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:24.636836 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:24.636842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:24.637179 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:24.637704 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:24.637720 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.638571 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.638575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:24.639804 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:24.639814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.640268 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.640271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:24.640407 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:24.640870 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:24.640885 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:24.642243 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:24.643723 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:24.643728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:24.643738 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:24.644037 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:24.644251 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:24.644253 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:24.644254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:24.644374 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:24.644663 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:24.644668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:24.644873 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 14:50:24.645016 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 14:50:24.645248 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
I20260926 14:50:24.647063 16602 heartbeater.cc:347] Connected to a master server at 127.12.68.254:41899
I20260926 14:50:24.647270 16602 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:24.647713 16602 heartbeater.cc:511] Master 127.12.68.254:41899 requested a full tablet report, sending...
I20260926 14:50:24.648731 16143 ts_manager.cc:194] Registered new tserver with Master: 3cc4b27924354087a306ffea096bacc4 (127.12.68.195:39217)
I20260926 14:50:24.649259 16143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:38359
I20260926 14:50:24.651358 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 14:50:24 dist-test-slave-24mn krb5kdc[16098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:50:24 dist-test-slave-24mn krb5kdc[16098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.193@KRBTEST.COM
I20260926 14:50:24.675026 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:24.666370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.12.68.193:45507)
0926 14:50:24.666664 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:24.666679 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:24.666704 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:50:24.666758 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:24.666762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:24.666960 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:24.667086 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:24.667717 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.668526 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.669332 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.669680 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.669768 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:24.671995 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:24.672003 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:24.672015 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:24.672037 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:24.673422 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:24.673804 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:24.673809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:24.673810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:24.673889 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:24.674122 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:24.674125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:24.674127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:24.674344 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:24.674533 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:24.674723 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 14:50:24.674873 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":104,"threads_started":1}
Sep 26 14:50:24 dist-test-slave-24mn krb5kdc[16098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:24.684563 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:24.677588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.12.68.254:41899)
0926 14:50:24.677721 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:24.677724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:24.677737 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:24.677834 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:24.677837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:24.677876 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:24.677936 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:24.678410 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.678904 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.679573 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:24.679768 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:24.679833 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:24.682000 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:24.682003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:24.682005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:24.682026 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:24.683172 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:24.683545 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:24.683548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:24.683549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:24.683625 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:24.683804 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:24.683806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:24.683807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:24.684057 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:24.684153 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:24.684329 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 14:50:24.684416 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 14:50:24.684895 16143 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57180
I20260926 14:50:24.685396 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 16205
I20260926 14:50:24.690706 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 16339
I20260926 14:50:24.695053 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 16473
I20260926 14:50:24.699393 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 16111
2026-09-26T14:50:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2018 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:24 dist-test-slave-24mn krb5kdc[16614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:24 dist-test-slave-24mn krb5kdc[16614](info): set up 2 sockets
Sep 26 14:50:24 dist-test-slave-24mn krb5kdc[16614](info): commencing operation
krb5kdc: starting...
W20260926 14:50:25.647688 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:50:25 dist-test-slave-24mn krb5kdc[16614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:50:25.755030 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:40625
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:40625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:25.859269 16627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:25.859532 16627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:25.859587 16627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:25.863543 16627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:25.863600 16627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:25.863626 16627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:25.863646 16627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:25.863667 16627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:25.868818 16627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:40625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:40625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:25.869791 16627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:25.870739 16627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:25.878464 16632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:25.878477 16633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:25.878463 16635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:25.878497 16627 server_base.cc:1034] running on GCE node
I20260926 14:50:25.879212 16627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:25.880313 16627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:25.881496 16627 hybrid_clock.cc:648] HybridClock initialized: now 1790434225881465 us; error 39 us; skew 500 ppm
Sep 26 14:50:25 dist-test-slave-24mn krb5kdc[16614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434225, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:25.893829 16627 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:50:25.894718 16627 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:25.894791 16627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:25.894991 16627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:25.897033 16627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "e8c4bb336ac44b4f809a6bec0d54142c"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-24mn"
server_key: "36e508c00f60a2f064b604da7bbb3ee0"
server_key_iv: "7307158515cd52fcfa26ad41d9254ce8"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.897559 16627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "e8c4bb336ac44b4f809a6bec0d54142c"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-24mn"
server_key: "36e508c00f60a2f064b604da7bbb3ee0"
server_key_iv: "7307158515cd52fcfa26ad41d9254ce8"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.901011 16627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:25.903133 16641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:25.904302 16627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:25.904412 16627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "e8c4bb336ac44b4f809a6bec0d54142c"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-24mn"
server_key: "36e508c00f60a2f064b604da7bbb3ee0"
server_key_iv: "7307158515cd52fcfa26ad41d9254ce8"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.904498 16627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:25.922128 16627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:25.924985 16627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:25.925146 16627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:25.931488 16627 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:40625
I20260926 14:50:25.931707 16693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:40625 every 8 connection(s)
I20260926 14:50:25.932103 16627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:25.934648 16694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:25.940047 16694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c: Bootstrap starting.
I20260926 14:50:25.940132 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 16627
I20260926 14:50:25.940254 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:25.940519 12563 external_mini_cluster.cc:1428] Setting key 1ccf22ea254a88da4e9c2ef0519114ca
I20260926 14:50:25.942515 16694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:25.943207 16694 log.cc:824] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:25.945223 16694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c: No bootstrap required, opened a new log
Sep 26 14:50:25 dist-test-slave-24mn krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:25.949154 16694 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c4bb336ac44b4f809a6bec0d54142c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40625 } }
I20260926 14:50:25.949425 16694 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:25.949491 16694 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c4bb336ac44b4f809a6bec0d54142c, State: Initialized, Role: FOLLOWER
I20260926 14:50:25.949895 16694 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c4bb336ac44b4f809a6bec0d54142c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40625 } }
I20260926 14:50:25.950021 16694 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:25.950084 16694 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:25.950176 16694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:25.950945 16694 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c4bb336ac44b4f809a6bec0d54142c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40625 } }
I20260926 14:50:25.951241 16694 leader_election.cc:302] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8c4bb336ac44b4f809a6bec0d54142c; no voters:
I20260926 14:50:25.951520 16694 leader_election.cc:288] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:25.951572 16699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:25.951737 16699 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [term 1 LEADER]: Becoming Leader. State: Replica: e8c4bb336ac44b4f809a6bec0d54142c, State: Running, Role: LEADER
I20260926 14:50:25.952126 16699 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c4bb336ac44b4f809a6bec0d54142c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40625 } }
I20260926 14:50:25.952785 16694 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:25.953328 16699 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8c4bb336ac44b4f809a6bec0d54142c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c4bb336ac44b4f809a6bec0d54142c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40625 } } }
I20260926 14:50:25.953444 16699 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:25.953802 16700 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8c4bb336ac44b4f809a6bec0d54142c. Latest consensus state: current_term: 1 leader_uuid: "e8c4bb336ac44b4f809a6bec0d54142c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c4bb336ac44b4f809a6bec0d54142c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40625 } } }
I20260926 14:50:25.953905 16700 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:25.954520 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:25.941754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.12.68.254:40625)
0926 14:50:25.942137 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:25.942152 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:25.942187 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:50:25.942861 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:25.942881 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:25.943208 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:25.943508 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:25.944245 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:25.945178 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:25.946312 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:25.946774 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:25.946891 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:25.949643 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:25.949649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:25.949661 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:25.949689 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:25.951298 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:25.951742 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:25.951748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:25.951754 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:25.951922 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:25.952216 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:25.952219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:25.952221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:25.952601 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:25.952783 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:25.953342 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 14:50:25.953649 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":117,"threads_started":1}
W20260926 14:50:25.957572 16714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:50:25.957647 16714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:50:25.957718 16705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:25.958511 16705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:25.963852 16705 catalog_manager.cc:1347] Generated new cluster ID: 3cf1858ec90642f5b8bd86d318c9a658
I20260926 14:50:25.963940 16705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:25.974752 16705 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:25.975605 16705 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:25.984431 16705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c: Generated new TSK 0
I20260926 14:50:25.985186 16705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:50:26.011412 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:40625
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:26.116938 16720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:26.117177 16720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:26.117218 16720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:26.121208 16720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:26.121263 16720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:26.121325 16720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:26.126416 16720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:40625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:26.127393 16720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:26.128376 16720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:26.136930 16725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:26.136989 16720 server_base.cc:1034] running on GCE node
W20260926 14:50:26.136947 16728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:26.136947 16726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:26.137533 16720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:26.138106 16720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:26.139252 16720 hybrid_clock.cc:648] HybridClock initialized: now 1790434226139236 us; error 27 us; skew 500 ppm
Sep 26 14:50:26 dist-test-slave-24mn krb5kdc[16614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:26.141799 16720 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:50:26.142429 16720 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:26.142493 16720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:26.142666 16720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:26.144614 16720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "356222fce93241a1a5ea0bb05055c3aa"
format_stamp: "Formatted at 2026-09-26 14:50:26 on dist-test-slave-24mn"
server_key: "4237ffd4ffc96c4ede6493518cbb9298"
server_key_iv: "22d2f124d9307eda2f91e57ad421acd7"
server_key_version: "encryptionkey@0"
I20260926 14:50:26.145116 16720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "356222fce93241a1a5ea0bb05055c3aa"
format_stamp: "Formatted at 2026-09-26 14:50:26 on dist-test-slave-24mn"
server_key: "4237ffd4ffc96c4ede6493518cbb9298"
server_key_iv: "22d2f124d9307eda2f91e57ad421acd7"
server_key_version: "encryptionkey@0"
I20260926 14:50:26.148485 16720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:50:26.150735 16734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:26.151979 16720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:26.152081 16720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "356222fce93241a1a5ea0bb05055c3aa"
format_stamp: "Formatted at 2026-09-26 14:50:26 on dist-test-slave-24mn"
server_key: "4237ffd4ffc96c4ede6493518cbb9298"
server_key_iv: "22d2f124d9307eda2f91e57ad421acd7"
server_key_version: "encryptionkey@0"
I20260926 14:50:26.152170 16720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:26.166193 16720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:26.168942 16720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:26.169102 16720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:26.169617 16720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:26.170375 16720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:26.170420 16720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:26.170470 16720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:26.170497 16720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:26.178546 16720 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:43071
I20260926 14:50:26.178565 16847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:43071 every 8 connection(s)
I20260926 14:50:26.179085 16720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:26.187103 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 16720
I20260926 14:50:26.187225 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:26.187525 12563 external_mini_cluster.cc:1428] Setting key 681dd5fed5e34664f44eb97ba691b8b2
Sep 26 14:50:26 dist-test-slave-24mn krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:26.192034 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.180705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:59995 (local address 127.12.68.254:40625)
0926 14:50:26.180837 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.180840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.181972 (+ 1132us) server_negotiation.cc:407] Connection header received
0926 14:50:26.182822 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.182825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.182888 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.182954 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:26.184108 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.184631 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.185723 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.185961 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.186047 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.188522 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:26.188525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:26.188527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:26.188551 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:26.190110 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.190583 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.190586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.190589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.190670 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.190976 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.190979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.190980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.191159 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:26.191242 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:26.191705 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 14:50:26.191849 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:50:26.192818 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.181083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40625 (local address 127.12.68.193:59995)
0926 14:50:26.181823 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:26.181867 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:26.182623 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:26.183127 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:26.183134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:26.183478 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:26.183968 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.183982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.184744 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.184747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.185611 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.185620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.186092 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.186096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.186248 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.186762 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:26.186778 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:26.188371 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:26.190223 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:26.190228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:26.190239 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:26.190488 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:26.190748 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:26.190752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:26.190754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:26.190892 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:26.191250 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:26.191256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:26.191487 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 14:50:26.191648 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 14:50:26.191982 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260926 14:50:26.194060 16848 heartbeater.cc:347] Connected to a master server at 127.12.68.254:40625
I20260926 14:50:26.194276 16848 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:26.194690 16848 heartbeater.cc:511] Master 127.12.68.254:40625 requested a full tablet report, sending...
I20260926 14:50:26.196106 16658 ts_manager.cc:194] Registered new tserver with Master: 356222fce93241a1a5ea0bb05055c3aa (127.12.68.193:43071)
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
I20260926 14:50:26.197206 16658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:59995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:50:26.212416 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:40625
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:26.318186 16853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:26.318432 16853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:26.318476 16853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:26.322350 16853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:26.322395 16853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:26.322450 16853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:26.327468 16853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:40625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:26.328413 16853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:26.329342 16853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:26.337934 16861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:26.337992 16859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:26.337992 16858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:26.338239 16853 server_base.cc:1034] running on GCE node
I20260926 14:50:26.338620 16853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:26.339206 16853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:26.340364 16853 hybrid_clock.cc:648] HybridClock initialized: now 1790434226340346 us; error 28 us; skew 500 ppm
Sep 26 14:50:26 dist-test-slave-24mn krb5kdc[16614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:26.342880 16853 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:50:26.343520 16853 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:26.343585 16853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:26.343770 16853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:26.345731 16853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "2cf9dde0761c48a197dc18aa606beb31"
format_stamp: "Formatted at 2026-09-26 14:50:26 on dist-test-slave-24mn"
server_key: "54c7b45ca9d93001669dff3f1e278151"
server_key_iv: "c245b906c5fd98f634efd390feeebf1a"
server_key_version: "encryptionkey@0"
I20260926 14:50:26.346240 16853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "2cf9dde0761c48a197dc18aa606beb31"
format_stamp: "Formatted at 2026-09-26 14:50:26 on dist-test-slave-24mn"
server_key: "54c7b45ca9d93001669dff3f1e278151"
server_key_iv: "c245b906c5fd98f634efd390feeebf1a"
server_key_version: "encryptionkey@0"
I20260926 14:50:26.349762 16853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:50:26.351938 16867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:26.353149 16853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:26.353251 16853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "2cf9dde0761c48a197dc18aa606beb31"
format_stamp: "Formatted at 2026-09-26 14:50:26 on dist-test-slave-24mn"
server_key: "54c7b45ca9d93001669dff3f1e278151"
server_key_iv: "c245b906c5fd98f634efd390feeebf1a"
server_key_version: "encryptionkey@0"
I20260926 14:50:26.353344 16853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:26.375334 16853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:26.378427 16853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:26.378595 16853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:26.379155 16853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:26.379979 16853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:26.380026 16853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:26.380077 16853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:26.380106 16853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:26.388242 16853 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:34201
I20260926 14:50:26.388271 16980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:34201 every 8 connection(s)
I20260926 14:50:26.388813 16853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:50:26 dist-test-slave-24mn krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:26.398279 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 16853
I20260926 14:50:26.398393 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:26.398633 12563 external_mini_cluster.cc:1428] Setting key 7eed9e7683f31a2b4cb7d515340dab7b
I20260926 14:50:26.401233 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.390368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:45421 (local address 127.12.68.254:40625)
0926 14:50:26.390527 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.390531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.391644 (+ 1113us) server_negotiation.cc:407] Connection header received
0926 14:50:26.392498 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.392501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.392548 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.392626 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:26.393716 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.394268 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.395367 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.395590 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.395672 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.397851 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:26.397854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:26.397856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:26.397881 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:26.399282 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.399759 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.399762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.399764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.399849 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.400211 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.400215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.400217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.400435 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:26.400539 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:26.400969 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 14:50:26.401083 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 14:50:26.401947 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.390772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40625 (local address 127.12.68.194:45421)
0926 14:50:26.391498 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:26.391535 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:26.392279 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:26.392768 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:26.392774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:26.393112 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:26.393556 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.393568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.394389 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.394391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.395246 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.395254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.395703 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.395706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.395870 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.396388 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:26.396404 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:26.397686 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:26.399376 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:26.399380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:26.399391 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:26.399637 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:26.399959 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:26.399963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:26.399964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:26.400119 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:26.400540 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:26.400545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:26.400760 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 14:50:26.400911 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 14:50:26.401142 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
I20260926 14:50:26.403123 16981 heartbeater.cc:347] Connected to a master server at 127.12.68.254:40625
I20260926 14:50:26.403331 16981 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:26.403717 16981 heartbeater.cc:511] Master 127.12.68.254:40625 requested a full tablet report, sending...
I20260926 14:50:26.404762 16658 ts_manager.cc:194] Registered new tserver with Master: 2cf9dde0761c48a197dc18aa606beb31 (127.12.68.194:34201)
I20260926 14:50:26.405503 16658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:45421
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:50:26.424394 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:40625
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:26.529246 16986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:26.529482 16986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:26.529529 16986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:26.533399 16986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:26.533445 16986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:26.533516 16986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:26.538584 16986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:40625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:26.539650 16986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:26.540625 16986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:26.549151 16991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:26.549170 16994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:26.549124 16992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:26.549203 16986 server_base.cc:1034] running on GCE node
I20260926 14:50:26.549710 16986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:26.550266 16986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:26.551422 16986 hybrid_clock.cc:648] HybridClock initialized: now 1790434226551400 us; error 33 us; skew 500 ppm
Sep 26 14:50:26 dist-test-slave-24mn krb5kdc[16614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:26.554087 16986 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:50:26.554783 16986 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:26.554844 16986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:26.555022 16986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:26.557039 16986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "506383dbcb2a40e2a0539b770c655678"
format_stamp: "Formatted at 2026-09-26 14:50:26 on dist-test-slave-24mn"
server_key: "2eb37c6f0ea1363d6ea17334962ddec3"
server_key_iv: "c55fc031221061b277d16054cedc8056"
server_key_version: "encryptionkey@0"
I20260926 14:50:26.557554 16986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "506383dbcb2a40e2a0539b770c655678"
format_stamp: "Formatted at 2026-09-26 14:50:26 on dist-test-slave-24mn"
server_key: "2eb37c6f0ea1363d6ea17334962ddec3"
server_key_iv: "c55fc031221061b277d16054cedc8056"
server_key_version: "encryptionkey@0"
I20260926 14:50:26.560916 16986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 14:50:26.563139 17000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:26.564319 16986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:26.564421 16986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "506383dbcb2a40e2a0539b770c655678"
format_stamp: "Formatted at 2026-09-26 14:50:26 on dist-test-slave-24mn"
server_key: "2eb37c6f0ea1363d6ea17334962ddec3"
server_key_iv: "c55fc031221061b277d16054cedc8056"
server_key_version: "encryptionkey@0"
I20260926 14:50:26.564505 16986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:26.601970 16986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:26.605098 16986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:26.605268 16986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:26.605819 16986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:26.606629 16986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:26.606676 16986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:26.606729 16986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:26.606755 16986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:26.614760 16986 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:38395
I20260926 14:50:26.614789 17113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:38395 every 8 connection(s)
I20260926 14:50:26.615312 16986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:26.619659 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 16986
I20260926 14:50:26.619756 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:26.620021 12563 external_mini_cluster.cc:1428] Setting key 04995645248b1c17448b591ebc07f4e9
Sep 26 14:50:26 dist-test-slave-24mn krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:26.628170 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.617006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:50211 (local address 127.12.68.254:40625)
0926 14:50:26.617157 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.617160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.618288 (+ 1128us) server_negotiation.cc:407] Connection header received
0926 14:50:26.619107 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.619110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.619158 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.619227 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:26.620490 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.621192 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.622565 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.622802 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.622878 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.625264 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:26.625267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:26.625269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:26.625291 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:26.626519 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.626946 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.626949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.626950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.627015 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.627265 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.627267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.627268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.627427 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:26.627494 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:26.627961 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 14:50:26.628051 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 14:50:26.628981 17116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.617374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40625 (local address 127.12.68.195:50211)
0926 14:50:26.618145 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:26.618188 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:26.618942 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:26.619375 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:26.619382 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:26.619782 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:26.620331 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.620353 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.621331 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.621334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.622457 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.622467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.622912 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.622915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.623051 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.623600 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:26.623646 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:26.625095 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:26.626625 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:26.626630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:26.626642 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:26.626872 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:26.627084 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:26.627086 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:26.627087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:26.627205 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:26.627495 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:26.627500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:26.627744 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 14:50:26.627941 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 14:50:26.628197 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260926 14:50:26.629958 17114 heartbeater.cc:347] Connected to a master server at 127.12.68.254:40625
I20260926 14:50:26.630188 17114 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:26.630595 17114 heartbeater.cc:511] Master 127.12.68.254:40625 requested a full tablet report, sending...
I20260926 14:50:26.631495 16658 ts_manager.cc:194] Registered new tserver with Master: 506383dbcb2a40e2a0539b770c655678 (127.12.68.195:38395)
I20260926 14:50:26.632082 16658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:50211
I20260926 14:50:26.633590 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:50:26.641611 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.635312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.12.68.254:40625)
0926 14:50:26.635441 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.635444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.635589 (+ 145us) server_negotiation.cc:407] Connection header received
0926 14:50:26.635691 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.635693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.635732 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.635786 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:26.636382 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.636872 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.637590 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.637779 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.637857 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.639000 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:26.639003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:26.639004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:26.639024 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:26.640371 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.640718 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.640720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.640722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.640786 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.640956 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.640958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.640959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.641150 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:26.641245 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:26.641394 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 14:50:26.641475 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:50:26.644220 16658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:26.646987 16658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:26.660766 17126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.655090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:43071 (local address 127.0.0.1:38266)
0926 14:50:26.656277 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:26.656288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:26.656358 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:26.656727 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:26.656729 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:26.656742 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:26.656931 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.656939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.658061 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.658063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.659231 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.659240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.660247 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.660251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.660357 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.660393 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 14:50:26.660508 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 14:50:26.660620 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"mutex_wait_us":18,"thread_start_us":83,"threads_started":1}
I20260926 14:50:26.660806 17129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.655640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.12.68.193:43071)
0926 14:50:26.656361 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.656367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.656410 (+ 43us) server_negotiation.cc:407] Connection header received
0926 14:50:26.656472 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.656476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.656618 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.656749 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:26.657060 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.657955 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.659350 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.659956 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.660050 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.660189 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:26.660557 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 14:50:26.660653 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":91,"threads_started":1}
I20260926 14:50:26.661310 17127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.654848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:38395 (local address 127.0.0.1:56254)
0926 14:50:26.655818 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:26.655872 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:26.656030 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:26.656681 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:26.656686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:26.656711 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:26.656931 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.656939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.658466 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.658469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.659722 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.659732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.660951 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.660954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.661039 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.661059 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:50:26.661126 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:50:26.661186 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":117,"spinlock_wait_cycles":7424,"thread_start_us":63,"threads_started":1}
I20260926 14:50:26.661383 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.655081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.12.68.195:38395)
0926 14:50:26.655535 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.655547 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.656235 (+ 688us) server_negotiation.cc:407] Connection header received
0926 14:50:26.656319 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.656324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.656549 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.656701 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:26.657083 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.658326 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.659846 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.660658 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.660748 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.660892 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:26.661157 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 14:50:26.661248 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":140,"threads_started":1}
I20260926 14:50:26.661463 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.654846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:34201 (local address 127.0.0.1:56190)
0926 14:50:26.655818 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:26.655872 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:26.656030 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:26.657463 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:26.657466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:26.657486 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:26.657568 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.657574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.658825 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.658828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.659953 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.659961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.661093 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.661096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.661185 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.661207 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:50:26.661274 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:50:26.661341 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":34,"spinlock_wait_cycles":23296,"thread_start_us":105,"threads_started":1}
I20260926 14:50:26.661917 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.655978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56190 (local address 127.12.68.194:34201)
0926 14:50:26.656974 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.656984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.657005 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:50:26.657072 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.657076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.657314 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.657461 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:26.657669 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.658692 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.660054 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.660606 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.660698 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.660837 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:26.661660 (+ 823us) server_negotiation.cc:299] Negotiation successful
0926 14:50:26.661771 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":85,"threads_started":1}
I20260926 14:50:26.663909 17048 tablet_service.cc:1467] Processing CreateTablet for tablet c491342ee9d24745bb7528f831533e23 (DEFAULT_TABLE table=test-table [id=bfe5d23c12f744179c84b5e9aae38fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:26.663938 16782 tablet_service.cc:1467] Processing CreateTablet for tablet c491342ee9d24745bb7528f831533e23 (DEFAULT_TABLE table=test-table [id=bfe5d23c12f744179c84b5e9aae38fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:26.664330 16915 tablet_service.cc:1467] Processing CreateTablet for tablet c491342ee9d24745bb7528f831533e23 (DEFAULT_TABLE table=test-table [id=bfe5d23c12f744179c84b5e9aae38fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:26.664983 17048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c491342ee9d24745bb7528f831533e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:26.664999 16782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c491342ee9d24745bb7528f831533e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:26.665192 16915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c491342ee9d24745bb7528f831533e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:26.670521 17131 tablet_bootstrap.cc:492] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678: Bootstrap starting.
I20260926 14:50:26.672443 17132 tablet_bootstrap.cc:492] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa: Bootstrap starting.
I20260926 14:50:26.672658 17131 tablet_bootstrap.cc:654] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:26.673163 17133 tablet_bootstrap.cc:492] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31: Bootstrap starting.
I20260926 14:50:26.673688 17131 log.cc:824] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:26.674476 17132 tablet_bootstrap.cc:654] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:26.675110 17133 tablet_bootstrap.cc:654] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:26.675329 17131 tablet_bootstrap.cc:492] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678: No bootstrap required, opened a new log
I20260926 14:50:26.675503 17131 ts_tablet_manager.cc:1397] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:50:26.675555 17132 log.cc:824] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:26.676323 17133 log.cc:824] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:26.677580 17132 tablet_bootstrap.cc:492] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa: No bootstrap required, opened a new log
I20260926 14:50:26.677798 17132 ts_tablet_manager.cc:1397] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:50:26.678056 17133 tablet_bootstrap.cc:492] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31: No bootstrap required, opened a new log
I20260926 14:50:26.678217 17133 ts_tablet_manager.cc:1397] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:50:26.678668 17131 raft_consensus.cc:348] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf9dde0761c48a197dc18aa606beb31" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 34201 } } peers { permanent_uuid: "506383dbcb2a40e2a0539b770c655678" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38395 } } peers { permanent_uuid: "356222fce93241a1a5ea0bb05055c3aa" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43071 } }
I20260926 14:50:26.678913 17131 raft_consensus.cc:374] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:26.678970 17131 raft_consensus.cc:729] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506383dbcb2a40e2a0539b770c655678, State: Initialized, Role: FOLLOWER
I20260926 14:50:26.679350 17131 consensus_queue.cc:260] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf9dde0761c48a197dc18aa606beb31" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 34201 } } peers { permanent_uuid: "506383dbcb2a40e2a0539b770c655678" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38395 } } peers { permanent_uuid: "356222fce93241a1a5ea0bb05055c3aa" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43071 } }
I20260926 14:50:26.679953 17114 heartbeater.cc:503] Master 127.12.68.254:40625 was elected leader, sending a full tablet report...
I20260926 14:50:26.681154 17131 ts_tablet_manager.cc:1428] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:50:26.681566 17132 raft_consensus.cc:348] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf9dde0761c48a197dc18aa606beb31" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 34201 } } peers { permanent_uuid: "506383dbcb2a40e2a0539b770c655678" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38395 } } peers { permanent_uuid: "356222fce93241a1a5ea0bb05055c3aa" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43071 } }
I20260926 14:50:26.681793 17132 raft_consensus.cc:374] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:26.681849 17132 raft_consensus.cc:729] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356222fce93241a1a5ea0bb05055c3aa, State: Initialized, Role: FOLLOWER
I20260926 14:50:26.682101 17133 raft_consensus.cc:348] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf9dde0761c48a197dc18aa606beb31" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 34201 } } peers { permanent_uuid: "506383dbcb2a40e2a0539b770c655678" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38395 } } peers { permanent_uuid: "356222fce93241a1a5ea0bb05055c3aa" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43071 } }
I20260926 14:50:26.682252 17132 consensus_queue.cc:260] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf9dde0761c48a197dc18aa606beb31" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 34201 } } peers { permanent_uuid: "506383dbcb2a40e2a0539b770c655678" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38395 } } peers { permanent_uuid: "356222fce93241a1a5ea0bb05055c3aa" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43071 } }
I20260926 14:50:26.682336 17133 raft_consensus.cc:374] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:26.682384 17133 raft_consensus.cc:729] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf9dde0761c48a197dc18aa606beb31, State: Initialized, Role: FOLLOWER
I20260926 14:50:26.682797 17133 consensus_queue.cc:260] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf9dde0761c48a197dc18aa606beb31" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 34201 } } peers { permanent_uuid: "506383dbcb2a40e2a0539b770c655678" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38395 } } peers { permanent_uuid: "356222fce93241a1a5ea0bb05055c3aa" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43071 } }
I20260926 14:50:26.682948 16848 heartbeater.cc:503] Master 127.12.68.254:40625 was elected leader, sending a full tablet report...
I20260926 14:50:26.683406 16981 heartbeater.cc:503] Master 127.12.68.254:40625 was elected leader, sending a full tablet report...
I20260926 14:50:26.683645 17133 ts_tablet_manager.cc:1428] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:50:26.684721 17132 ts_tablet_manager.cc:1428] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260926 14:50:26.866103 17115 tablet.cc:2369] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:50:26.889837 16982 tablet.cc:2369] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:26.893946 17138 raft_consensus.cc:482] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:26.894130 17138 raft_consensus.cc:504] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf9dde0761c48a197dc18aa606beb31" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 34201 } } peers { permanent_uuid: "506383dbcb2a40e2a0539b770c655678" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38395 } } peers { permanent_uuid: "356222fce93241a1a5ea0bb05055c3aa" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43071 } }
I20260926 14:50:26.895220 17138 leader_election.cc:288] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cf9dde0761c48a197dc18aa606beb31 (127.12.68.194:34201), 506383dbcb2a40e2a0539b770c655678 (127.12.68.195:38395)
I20260926 14:50:26.899080 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.895599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:46717 (local address 127.12.68.195:38395)
0926 14:50:26.895784 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.895789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.895914 (+ 125us) server_negotiation.cc:407] Connection header received
0926 14:50:26.896117 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.896120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.896161 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.896220 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:26.896530 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.897018 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.898134 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.898523 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.898625 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.898676 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:26.898876 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 14:50:26.898950 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 14:50:26.899108 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.895448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:34201 (local address 127.12.68.193:53847)
0926 14:50:26.895733 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:26.895749 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:26.895905 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:26.896265 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:26.896267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:26.896303 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:26.896443 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.896448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.897125 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.897128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.898029 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.898038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.898641 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.898644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.898751 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.898768 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 14:50:26.898882 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 14:50:26.898955 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":86,"spinlock_wait_cycles":28800,"thread_start_us":81,"threads_started":1}
I20260926 14:50:26.899147 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.895542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:53847 (local address 127.12.68.194:34201)
0926 14:50:26.895739 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.895744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.895824 (+ 80us) server_negotiation.cc:407] Connection header received
0926 14:50:26.896034 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.896037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.896087 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.896132 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:26.896551 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.897020 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.898148 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.898534 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.898636 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.898686 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:26.898901 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 14:50:26.898997 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260926 14:50:26.899434 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.895473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:38395 (local address 127.12.68.193:46717)
0926 14:50:26.895821 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:26.895834 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:26.895971 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:26.896246 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:26.896249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:26.896264 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:26.896435 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.896441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.897120 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.897123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.898020 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.898030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.898630 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.898633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.898729 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.898752 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:50:26.899219 (+ 467us) client_negotiation.cc:240] Negotiation successful
0926 14:50:26.899293 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":41,"spinlock_wait_cycles":2816,"thread_start_us":90,"threads_started":1}
I20260926 14:50:26.899780 16935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c491342ee9d24745bb7528f831533e23" candidate_uuid: "356222fce93241a1a5ea0bb05055c3aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf9dde0761c48a197dc18aa606beb31" is_pre_election: true
I20260926 14:50:26.900125 16935 raft_consensus.cc:2454] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 356222fce93241a1a5ea0bb05055c3aa in term 0.
I20260926 14:50:26.900156 17068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c491342ee9d24745bb7528f831533e23" candidate_uuid: "356222fce93241a1a5ea0bb05055c3aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506383dbcb2a40e2a0539b770c655678" is_pre_election: true
I20260926 14:50:26.900425 17068 raft_consensus.cc:2454] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 356222fce93241a1a5ea0bb05055c3aa in term 0.
I20260926 14:50:26.900594 16735 leader_election.cc:302] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cf9dde0761c48a197dc18aa606beb31, 356222fce93241a1a5ea0bb05055c3aa; no voters:
I20260926 14:50:26.900894 17138 raft_consensus.cc:2781] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:26.900970 17138 raft_consensus.cc:482] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:26.901022 17138 raft_consensus.cc:3037] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:26.901969 17138 raft_consensus.cc:504] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf9dde0761c48a197dc18aa606beb31" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 34201 } } peers { permanent_uuid: "506383dbcb2a40e2a0539b770c655678" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38395 } } peers { permanent_uuid: "356222fce93241a1a5ea0bb05055c3aa" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43071 } }
I20260926 14:50:26.902307 17138 leader_election.cc:288] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [CANDIDATE]: Term 1 election: Requested vote from peers 2cf9dde0761c48a197dc18aa606beb31 (127.12.68.194:34201), 506383dbcb2a40e2a0539b770c655678 (127.12.68.195:38395)
I20260926 14:50:26.902611 16935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c491342ee9d24745bb7528f831533e23" candidate_uuid: "356222fce93241a1a5ea0bb05055c3aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf9dde0761c48a197dc18aa606beb31"
I20260926 14:50:26.902611 17068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c491342ee9d24745bb7528f831533e23" candidate_uuid: "356222fce93241a1a5ea0bb05055c3aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506383dbcb2a40e2a0539b770c655678"
I20260926 14:50:26.902726 16935 raft_consensus.cc:3037] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:26.902726 17068 raft_consensus.cc:3037] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:26.903930 17068 raft_consensus.cc:2454] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 356222fce93241a1a5ea0bb05055c3aa in term 1.
I20260926 14:50:26.903930 16935 raft_consensus.cc:2454] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 356222fce93241a1a5ea0bb05055c3aa in term 1.
I20260926 14:50:26.904201 16735 leader_election.cc:302] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cf9dde0761c48a197dc18aa606beb31, 356222fce93241a1a5ea0bb05055c3aa; no voters:
I20260926 14:50:26.904368 17138 raft_consensus.cc:2781] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:26.904569 17138 raft_consensus.cc:686] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [term 1 LEADER]: Becoming Leader. State: Replica: 356222fce93241a1a5ea0bb05055c3aa, State: Running, Role: LEADER
I20260926 14:50:26.904872 17138 consensus_queue.cc:237] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf9dde0761c48a197dc18aa606beb31" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 34201 } } peers { permanent_uuid: "506383dbcb2a40e2a0539b770c655678" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38395 } } peers { permanent_uuid: "356222fce93241a1a5ea0bb05055c3aa" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43071 } }
I20260926 14:50:26.907786 16657 catalog_manager.cc:5495] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 356222fce93241a1a5ea0bb05055c3aa (127.12.68.193). New cstate: current_term: 1 leader_uuid: "356222fce93241a1a5ea0bb05055c3aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf9dde0761c48a197dc18aa606beb31" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 34201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506383dbcb2a40e2a0539b770c655678" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "356222fce93241a1a5ea0bb05055c3aa" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43071 } health_report { overall_health: HEALTHY } } }
W20260926 14:50:26.930148 16849 tablet.cc:2369] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:26.954166 17129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.950963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38278 (local address 127.12.68.193:43071)
0926 14:50:26.951066 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.951069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.951082 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:26.951167 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.951169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.951222 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.951306 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:26.951489 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.952009 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.952918 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.953132 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.953204 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.953496 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:26.953893 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:26.953978 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:26.954019 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 14:50:26.954065 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:50:26.959933 17068 raft_consensus.cc:1261] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678 [term 1 FOLLOWER]: Refusing update from remote peer 356222fce93241a1a5ea0bb05055c3aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:26.959933 16935 raft_consensus.cc:1261] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31 [term 1 FOLLOWER]: Refusing update from remote peer 356222fce93241a1a5ea0bb05055c3aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:26.960649 17138 consensus_queue.cc:1035] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "506383dbcb2a40e2a0539b770c655678" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 38395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:26.960851 17142 consensus_queue.cc:1035] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf9dde0761c48a197dc18aa606beb31" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 34201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:26.967060 17145 mvcc.cc:205] Tried to move back new op lower bound from 7333618593620385792 to 7333618593404760064. Current Snapshot: MvccSnapshot[applied={T|T < 7333618593620385792}]
I20260926 14:50:26.967567 17147 mvcc.cc:205] Tried to move back new op lower bound from 7333618593620385792 to 7333618593404760064. Current Snapshot: MvccSnapshot[applied={T|T < 7333618593620385792}]
I20260926 14:50:26.970209 17146 mvcc.cc:205] Tried to move back new op lower bound from 7333618593620385792 to 7333618593404760064. Current Snapshot: MvccSnapshot[applied={T|T < 7333618593620385792}]
I20260926 14:50:26.975183 16657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:26.975322 16657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:26.977435 16657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8c4bb336ac44b4f809a6bec0d54142c: Sending DeleteTablet for 3 replicas of tablet c491342ee9d24745bb7528f831533e23
I20260926 14:50:26.978127 17048 tablet_service.cc:1514] Processing DeleteTablet for tablet c491342ee9d24745bb7528f831533e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:26 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:56254
I20260926 14:50:26.978121 16915 tablet_service.cc:1514] Processing DeleteTablet for tablet c491342ee9d24745bb7528f831533e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:26 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:56190
I20260926 14:50:26.978322 16782 tablet_service.cc:1514] Processing DeleteTablet for tablet c491342ee9d24745bb7528f831533e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:26 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:38266
I20260926 14:50:26.978610 17154 tablet_replica.cc:330] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678: stopping tablet replica
I20260926 14:50:26.978771 17156 tablet_replica.cc:330] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa: stopping tablet replica
I20260926 14:50:26.978947 17154 raft_consensus.cc:2229] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:26.979041 17156 raft_consensus.cc:2229] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:26.979305 17154 raft_consensus.cc:2258] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:26.979417 17156 raft_consensus.cc:2258] T c491342ee9d24745bb7528f831533e23 P 356222fce93241a1a5ea0bb05055c3aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:26.979421 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 16720
I20260926 14:50:26.979665 17155 tablet_replica.cc:330] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31: stopping tablet replica
I20260926 14:50:26.979982 17155 raft_consensus.cc:2229] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:26.980252 17155 raft_consensus.cc:2258] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:26.981356 17154 ts_tablet_manager.cc:1905] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:26.981494 17155 ts_tablet_manager.cc:1905] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:26.983472 17154 ts_tablet_manager.cc:1918] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:26.983551 17154 log.cc:1196] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/c491342ee9d24745bb7528f831533e23
I20260926 14:50:26.983742 17155 ts_tablet_manager.cc:1918] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:26.983814 17155 log.cc:1196] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/c491342ee9d24745bb7528f831533e23
I20260926 14:50:26.983812 17154 ts_tablet_manager.cc:1939] T c491342ee9d24745bb7528f831533e23 P 506383dbcb2a40e2a0539b770c655678: Deleting consensus metadata
I20260926 14:50:26.984081 17155 ts_tablet_manager.cc:1939] T c491342ee9d24745bb7528f831533e23 P 2cf9dde0761c48a197dc18aa606beb31: Deleting consensus metadata
I20260926 14:50:26.984858 16644 catalog_manager.cc:4841] TS 506383dbcb2a40e2a0539b770c655678 (127.12.68.195:38395): tablet c491342ee9d24745bb7528f831533e23 (table test-table [id=bfe5d23c12f744179c84b5e9aae38fa0]) successfully deleted
I20260926 14:50:26.984970 16642 catalog_manager.cc:4841] TS 2cf9dde0761c48a197dc18aa606beb31 (127.12.68.194:34201): tablet c491342ee9d24745bb7528f831533e23 (table test-table [id=bfe5d23c12f744179c84b5e9aae38fa0]) successfully deleted
W20260926 14:50:26.988162 16645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:43071 (error 108)
I20260926 14:50:26.988341 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 16853
I20260926 14:50:26.990573 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.990341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:43071 (local address 127.0.0.1:38294)
0926 14:50:26.990452 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:26.990504 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 14:50:26.990712 16645 catalog_manager.cc:4584] TS 356222fce93241a1a5ea0bb05055c3aa (127.12.68.193:43071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c491342ee9d24745bb7528f831533e23: Network error: Client connection negotiation failed: client connection to 127.12.68.193:43071: connect: Connection refused (error 111)
I20260926 14:50:26.993171 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 16986
I20260926 14:50:26.997781 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 16627
2026-09-26T14:50:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2298 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T14:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:27Z Disabled control of system clock
I20260926 14:50:27.024664 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:43907
--webserver_interface=127.12.68.254
--webserver_port=0
--builtin_ntp_servers=127.12.68.212:44317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:43907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 14:50:27.129484 17161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:27.129743 17161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:27.129779 17161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:27.133669 17161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:27.133718 17161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:27.133731 17161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:27.133746 17161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:27.133759 17161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:27.138329 17161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:44317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:43907
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:43907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:27.139349 17161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:27.140654 17161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:27.148561 17167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:27.148576 17166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:27.148875 17161 server_base.cc:1034] running on GCE node
W20260926 14:50:27.148895 17169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:27.149408 17161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:27.150467 17161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:27.151633 17161 hybrid_clock.cc:648] HybridClock initialized: now 1790434227151616 us; error 30 us; skew 500 ppm
I20260926 14:50:27.153487 17161 webserver.cc:460] Webserver started at http://127.12.68.254:34695/ using document root <none> and password file <none>
I20260926 14:50:27.154038 17161 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:27.154103 17161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:27.154284 17161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:27.156299 17161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "35d3a627f7f44a17b8870fb452183909"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-24mn"
server_key: "18201332478fbaa6c08f4d269547b2aa"
server_key_iv: "8d2a728a60716ddcafb3081cf8e1b26c"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.156821 17161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "35d3a627f7f44a17b8870fb452183909"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-24mn"
server_key: "18201332478fbaa6c08f4d269547b2aa"
server_key_iv: "8d2a728a60716ddcafb3081cf8e1b26c"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.160427 17161 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:27.162680 17175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.163818 17161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:27.163959 17161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "35d3a627f7f44a17b8870fb452183909"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-24mn"
server_key: "18201332478fbaa6c08f4d269547b2aa"
server_key_iv: "8d2a728a60716ddcafb3081cf8e1b26c"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.164055 17161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:27.168298 17161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:27.168876 17161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:27.169031 17161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:27.175201 17161 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:43907
I20260926 14:50:27.175215 17227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:43907 every 8 connection(s)
I20260926 14:50:27.176106 17161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:27.179070 17228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:27.179373 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 17161
I20260926 14:50:27.179459 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:27.179703 12563 external_mini_cluster.cc:1428] Setting key 320a39186da5908ceaa5670cbf6d9880
I20260926 14:50:27.184862 17228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909: Bootstrap starting.
I20260926 14:50:27.185035 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.181047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.12.68.254:43907)
0926 14:50:27.181624 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.181639 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.181672 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:50:27.182315 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.182333 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.182676 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.182953 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.183766 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.183771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.183782 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:27.183792 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.183855 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:27.183873 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.184027 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.184132 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.184356 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":112,"threads_started":1}
I20260926 14:50:27.187203 17228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:27.187863 17228 log.cc:824] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:27.189618 17228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909: No bootstrap required, opened a new log
I20260926 14:50:27.192909 17228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d3a627f7f44a17b8870fb452183909" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43907 } }
I20260926 14:50:27.193120 17228 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:27.193182 17228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d3a627f7f44a17b8870fb452183909, State: Initialized, Role: FOLLOWER
I20260926 14:50:27.193554 17228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d3a627f7f44a17b8870fb452183909" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43907 } }
I20260926 14:50:27.193676 17228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:27.193738 17228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:27.193832 17228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:27.194618 17228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d3a627f7f44a17b8870fb452183909" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43907 } }
I20260926 14:50:27.194921 17228 leader_election.cc:302] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35d3a627f7f44a17b8870fb452183909; no voters:
I20260926 14:50:27.195199 17228 leader_election.cc:288] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:27.195282 17233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:27.195519 17233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [term 1 LEADER]: Becoming Leader. State: Replica: 35d3a627f7f44a17b8870fb452183909, State: Running, Role: LEADER
I20260926 14:50:27.195958 17233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d3a627f7f44a17b8870fb452183909" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43907 } }
I20260926 14:50:27.196307 17228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:27.197373 17235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35d3a627f7f44a17b8870fb452183909. Latest consensus state: current_term: 1 leader_uuid: "35d3a627f7f44a17b8870fb452183909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d3a627f7f44a17b8870fb452183909" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43907 } } }
I20260926 14:50:27.197494 17235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:27.197870 17242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:27.197846 17234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35d3a627f7f44a17b8870fb452183909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d3a627f7f44a17b8870fb452183909" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43907 } } }
I20260926 14:50:27.197975 17234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:27.200623 17242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:27.205410 17242 catalog_manager.cc:1347] Generated new cluster ID: f57c3fa8262342cebb794a470b05873b
I20260926 14:50:27.205468 17242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:27.227664 17242 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:27.228066 17242 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:27.237046 17242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909: Generated new TSK 0
I20260926 14:50:27.237880 17242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:50:27.240345 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43907
--builtin_ntp_servers=127.12.68.212:44317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 14:50:27.346107 17252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:27.346330 17252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:27.346369 17252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:27.350335 17252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:27.350397 17252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:27.350478 17252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:27.355274 17252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:44317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:27.356225 17252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:27.357468 17252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:27.366242 17257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:27.366267 17260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:27.366333 17252 server_base.cc:1034] running on GCE node
W20260926 14:50:27.366248 17258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:27.366868 17252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:27.367446 17252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:27.368625 17252 hybrid_clock.cc:648] HybridClock initialized: now 1790434227368607 us; error 30 us; skew 500 ppm
I20260926 14:50:27.370288 17252 webserver.cc:460] Webserver started at http://127.12.68.193:38023/ using document root <none> and password file <none>
I20260926 14:50:27.370805 17252 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:27.370864 17252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:27.371033 17252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:27.372951 17252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "3026ec8085324ea983bd05c67d7172d1"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-24mn"
server_key: "5dd0c9a75b9097b8de52fd2b0f333852"
server_key_iv: "42dd0275d4fb8b2571a621492210d9c4"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.373446 17252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "3026ec8085324ea983bd05c67d7172d1"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-24mn"
server_key: "5dd0c9a75b9097b8de52fd2b0f333852"
server_key_iv: "42dd0275d4fb8b2571a621492210d9c4"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.376785 17252 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:27.378993 17266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.380151 17252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:27.380247 17252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "3026ec8085324ea983bd05c67d7172d1"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-24mn"
server_key: "5dd0c9a75b9097b8de52fd2b0f333852"
server_key_iv: "42dd0275d4fb8b2571a621492210d9c4"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.380338 17252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:27.384455 17252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:27.384979 17252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:27.385120 17252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:27.385589 17252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:27.386397 17252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:27.386440 17252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.386487 17252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:27.386513 17252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.394080 17252 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:44697
I20260926 14:50:27.394109 17379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:44697 every 8 connection(s)
I20260926 14:50:27.394863 17252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:27.395919 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 17252
I20260926 14:50:27.396030 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:27.396294 12563 external_mini_cluster.cc:1428] Setting key 77fae38d71babd92f478d70125191278
I20260926 14:50:27.398102 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43907
--builtin_ntp_servers=127.12.68.212:44317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 14:50:27.400883 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.396627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:34517 (local address 127.12.68.254:43907)
0926 14:50:27.396739 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.396743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.397970 (+ 1227us) server_negotiation.cc:407] Connection header received
0926 14:50:27.399269 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.399272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.399315 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.399388 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.400452 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.400455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.400458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:27.400470 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.400491 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:27.400499 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.400572 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.400691 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.400733 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:50:27.401705 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.397092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43907 (local address 127.12.68.193:34517)
0926 14:50:27.397830 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:27.397877 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:27.399074 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:27.399560 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:27.399567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:27.399653 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:27.400209 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:27.400211 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:27.400248 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:27.400319 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:27.400565 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:27.400573 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:27.400579 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:50:27.400724 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 14:50:27.400946 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":116,"threads_started":1}
I20260926 14:50:27.402861 17380 heartbeater.cc:347] Connected to a master server at 127.12.68.254:43907
I20260926 14:50:27.403102 17380 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:27.403553 17380 heartbeater.cc:511] Master 127.12.68.254:43907 requested a full tablet report, sending...
I20260926 14:50:27.405045 17192 ts_manager.cc:194] Registered new tserver with Master: 3026ec8085324ea983bd05c67d7172d1 (127.12.68.193:44697)
W20260926 14:50:27.504024 17383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:27.504257 17383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:27.504293 17383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:27.508168 17383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:27.508221 17383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:27.508275 17383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:27.512800 17383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:44317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:27.513742 17383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:27.514991 17383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:27.523694 17388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:27.523692 17389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:27.523805 17383 server_base.cc:1034] running on GCE node
W20260926 14:50:27.523759 17391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:27.524312 17383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:27.524868 17383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:27.526022 17383 hybrid_clock.cc:648] HybridClock initialized: now 1790434227526006 us; error 28 us; skew 500 ppm
I20260926 14:50:27.527678 17383 webserver.cc:460] Webserver started at http://127.12.68.194:43477/ using document root <none> and password file <none>
I20260926 14:50:27.528252 17383 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:27.528317 17383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:27.528496 17383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:27.530369 17383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "11049d7ee6344fe9986736ed99af9a93"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-24mn"
server_key: "183dab1a3a72e14ae59e8d92e8ed537f"
server_key_iv: "9965181f224c2d801c302a5dadb67e08"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.530861 17383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "11049d7ee6344fe9986736ed99af9a93"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-24mn"
server_key: "183dab1a3a72e14ae59e8d92e8ed537f"
server_key_iv: "9965181f224c2d801c302a5dadb67e08"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.534121 17383 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:50:27.536264 17397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.537312 17383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:27.537406 17383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "11049d7ee6344fe9986736ed99af9a93"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-24mn"
server_key: "183dab1a3a72e14ae59e8d92e8ed537f"
server_key_iv: "9965181f224c2d801c302a5dadb67e08"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.537489 17383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:27.541594 17383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:27.542104 17383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:27.542251 17383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:27.542714 17383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:27.543493 17383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:27.543537 17383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.543581 17383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:27.543609 17383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.551538 17383 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:43429
I20260926 14:50:27.551569 17510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:43429 every 8 connection(s)
I20260926 14:50:27.552338 17383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:27.553228 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 17383
I20260926 14:50:27.553354 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:27.553622 12563 external_mini_cluster.cc:1428] Setting key 321781301058cb60cfb4a7b8c2c77955
I20260926 14:50:27.555500 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43907
--builtin_ntp_servers=127.12.68.212:44317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 14:50:27.558045 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.554087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:47737 (local address 127.12.68.254:43907)
0926 14:50:27.554229 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.554233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.555273 (+ 1040us) server_negotiation.cc:407] Connection header received
0926 14:50:27.556555 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.556559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.556600 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.556672 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.557649 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.557652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.557655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:27.557667 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.557704 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:27.557714 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.557788 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.557913 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.557948 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:50:27.559021 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.554478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43907 (local address 127.12.68.194:47737)
0926 14:50:27.555146 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:27.555182 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:27.556366 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:27.556827 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:27.556833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:27.556915 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:27.557415 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:27.557417 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:27.557451 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:27.557511 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:27.557789 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:27.557795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:27.557801 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:50:27.557947 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 14:50:27.558158 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
I20260926 14:50:27.559984 17511 heartbeater.cc:347] Connected to a master server at 127.12.68.254:43907
I20260926 14:50:27.560212 17511 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:27.560640 17511 heartbeater.cc:511] Master 127.12.68.254:43907 requested a full tablet report, sending...
I20260926 14:50:27.561725 17192 ts_manager.cc:194] Registered new tserver with Master: 11049d7ee6344fe9986736ed99af9a93 (127.12.68.194:43429)
W20260926 14:50:27.658128 17514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:27.658351 17514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:27.658402 17514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:27.662274 17514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:27.662319 17514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:27.662392 17514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:27.667377 17514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:44317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:27.668387 17514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:27.669637 17514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:27.678262 17519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:27.678272 17520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:27.678272 17522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:27.678320 17514 server_base.cc:1034] running on GCE node
I20260926 14:50:27.678874 17514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:27.679416 17514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:27.680558 17514 hybrid_clock.cc:648] HybridClock initialized: now 1790434227680536 us; error 33 us; skew 500 ppm
I20260926 14:50:27.682161 17514 webserver.cc:460] Webserver started at http://127.12.68.195:35543/ using document root <none> and password file <none>
I20260926 14:50:27.682655 17514 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:27.682723 17514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:27.682929 17514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:27.684899 17514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "2d3a4d7beda44b2cbe1a54abe8177931"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-24mn"
server_key: "34f6f8f1c392bd7c101911b09a193dbf"
server_key_iv: "08b84ff06a89c3b304930e618d3a9619"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.685390 17514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "2d3a4d7beda44b2cbe1a54abe8177931"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-24mn"
server_key: "34f6f8f1c392bd7c101911b09a193dbf"
server_key_iv: "08b84ff06a89c3b304930e618d3a9619"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.688666 17514 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:27.691004 17528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.692160 17514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:27.692258 17514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "2d3a4d7beda44b2cbe1a54abe8177931"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-24mn"
server_key: "34f6f8f1c392bd7c101911b09a193dbf"
server_key_iv: "08b84ff06a89c3b304930e618d3a9619"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.692348 17514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:27.696511 17514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:27.697326 17514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:27.697472 17514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:27.697964 17514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:27.698860 17514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:27.698904 17514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.698949 17514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:27.698977 17514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.706930 17514 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:43833
I20260926 14:50:27.706961 17641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:43833 every 8 connection(s)
I20260926 14:50:27.707712 17514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:27.711004 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 17514
I20260926 14:50:27.711102 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:27.711331 12563 external_mini_cluster.cc:1428] Setting key 1edcd2dbe9b897563a333b9ab0331795
I20260926 14:50:27.713346 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.709414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:43495 (local address 127.12.68.254:43907)
0926 14:50:27.709541 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.709544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.710539 (+ 995us) server_negotiation.cc:407] Connection header received
0926 14:50:27.711822 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.711825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.711868 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.711960 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.712914 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.712917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.712919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:27.712930 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.712950 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:27.712959 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.713029 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.713189 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.713227 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:50:27.714135 17644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.709756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43907 (local address 127.12.68.195:43495)
0926 14:50:27.710395 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:27.710432 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:27.711636 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:27.712105 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:27.712111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:27.712194 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:27.712683 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:27.712684 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:27.712722 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:27.712790 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:27.713048 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:27.713057 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:27.713063 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:50:27.713227 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 14:50:27.713427 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260926 14:50:27.715073 17642 heartbeater.cc:347] Connected to a master server at 127.12.68.254:43907
I20260926 14:50:27.715260 17642 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:27.715616 17642 heartbeater.cc:511] Master 127.12.68.254:43907 requested a full tablet report, sending...
I20260926 14:50:27.716495 17192 ts_manager.cc:194] Registered new tserver with Master: 2d3a4d7beda44b2cbe1a54abe8177931 (127.12.68.195:43833)
I20260926 14:50:27.724576 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:50:27.727428 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.726196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.12.68.254:43907)
0926 14:50:27.726297 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.726301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.726452 (+ 151us) server_negotiation.cc:407] Connection header received
0926 14:50:27.726541 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.726543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.726568 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.726625 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.727185 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.727187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.727189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:27.727195 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.727207 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:27.727213 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.727264 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.727314 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.727342 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:50:27.729681 17192 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:27.732436 17192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:27.742506 17650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.739611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:43833 (local address 127.0.0.1:34076)
0926 14:50:27.739968 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:27.740012 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:27.740101 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:27.740951 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:27.740957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:27.741031 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:27.741534 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:27.741536 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:27.741555 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:27.741620 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:27.742178 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:27.742187 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:27.742193 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:50:27.742312 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 14:50:27.742376 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260926 14:50:27.742684 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.739862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44697 (local address 127.0.0.1:49498)
0926 14:50:27.740274 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:27.740286 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:27.740342 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:27.741132 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:27.741135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:27.741149 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:27.741992 (+ 843us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:27.741993 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:27.742003 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:27.742024 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:27.742488 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:27.742491 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:27.742494 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:27.742548 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 14:50:27.742579 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":54,"threads_started":1}
I20260926 14:50:27.742877 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.740322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:43429 (local address 127.0.0.1:52988)
0926 14:50:27.740845 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:27.740858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:27.740922 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:27.741518 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:27.741522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:27.741536 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:27.741834 (+ 298us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:27.741836 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:27.741846 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:27.741887 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:27.742687 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:27.742691 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:27.742694 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:27.742747 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 14:50:27.742790 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":60,"threads_started":1}
I20260926 14:50:27.742537 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.739738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34076 (local address 127.12.68.195:43833)
0926 14:50:27.740175 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.740187 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.740212 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:50:27.740269 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.740275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.740452 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.741296 (+ 844us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.741878 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.741884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.741895 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:27.741904 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.742020 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:27.742048 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.742182 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.742325 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.742392 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":163,"threads_started":1}
I20260926 14:50:27.743178 17654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.740367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.12.68.193:44697)
0926 14:50:27.740681 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.740693 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.740716 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:50:27.740768 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.740817 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.740998 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.741776 (+ 778us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.742308 (+ 532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.742313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.742324 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:27.742333 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.742369 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:27.742390 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.742926 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.742976 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.743043 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":112,"threads_started":1}
I20260926 14:50:27.743270 17655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.740625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52988 (local address 127.12.68.194:43429)
0926 14:50:27.741144 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.741154 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.741174 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:50:27.741220 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.741225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.741377 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.741711 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.742504 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.742508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.742519 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:27.742528 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.742565 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:27.742590 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.743026 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.743069 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.743129 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":103,"threads_started":1}
I20260926 14:50:27.745784 17445 tablet_service.cc:1467] Processing CreateTablet for tablet c1e83c9200bd42179f3c70435f0858cb (DEFAULT_TABLE table=test-table [id=d7e74a0e842d4b529cf12f8fd1958247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:27.745792 17314 tablet_service.cc:1467] Processing CreateTablet for tablet c1e83c9200bd42179f3c70435f0858cb (DEFAULT_TABLE table=test-table [id=d7e74a0e842d4b529cf12f8fd1958247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:27.745790 17576 tablet_service.cc:1467] Processing CreateTablet for tablet c1e83c9200bd42179f3c70435f0858cb (DEFAULT_TABLE table=test-table [id=d7e74a0e842d4b529cf12f8fd1958247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:27.746688 17445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e83c9200bd42179f3c70435f0858cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:27.746757 17576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e83c9200bd42179f3c70435f0858cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:27.746757 17314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e83c9200bd42179f3c70435f0858cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:27.752295 17656 tablet_bootstrap.cc:492] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93: Bootstrap starting.
I20260926 14:50:27.753793 17657 tablet_bootstrap.cc:492] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1: Bootstrap starting.
I20260926 14:50:27.754658 17656 tablet_bootstrap.cc:654] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:27.755571 17657 tablet_bootstrap.cc:654] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:27.755650 17656 log.cc:824] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:27.756188 17658 tablet_bootstrap.cc:492] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931: Bootstrap starting.
I20260926 14:50:27.756377 17657 log.cc:824] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:27.757465 17656 tablet_bootstrap.cc:492] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93: No bootstrap required, opened a new log
I20260926 14:50:27.757654 17656 ts_tablet_manager.cc:1397] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:50:27.758100 17658 tablet_bootstrap.cc:654] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:27.758206 17657 tablet_bootstrap.cc:492] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1: No bootstrap required, opened a new log
I20260926 14:50:27.758381 17657 ts_tablet_manager.cc:1397] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:50:27.758870 17658 log.cc:824] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:27.760694 17658 tablet_bootstrap.cc:492] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931: No bootstrap required, opened a new log
I20260926 14:50:27.760864 17658 ts_tablet_manager.cc:1397] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:50:27.761344 17656 raft_consensus.cc:348] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 43833 } } peers { permanent_uuid: "11049d7ee6344fe9986736ed99af9a93" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43429 } } peers { permanent_uuid: "3026ec8085324ea983bd05c67d7172d1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44697 } }
I20260926 14:50:27.761550 17656 raft_consensus.cc:374] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:27.761616 17656 raft_consensus.cc:729] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11049d7ee6344fe9986736ed99af9a93, State: Initialized, Role: FOLLOWER
I20260926 14:50:27.761999 17656 consensus_queue.cc:260] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 43833 } } peers { permanent_uuid: "11049d7ee6344fe9986736ed99af9a93" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43429 } } peers { permanent_uuid: "3026ec8085324ea983bd05c67d7172d1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44697 } }
I20260926 14:50:27.762342 17657 raft_consensus.cc:348] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 43833 } } peers { permanent_uuid: "11049d7ee6344fe9986736ed99af9a93" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43429 } } peers { permanent_uuid: "3026ec8085324ea983bd05c67d7172d1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44697 } }
I20260926 14:50:27.762557 17511 heartbeater.cc:503] Master 127.12.68.254:43907 was elected leader, sending a full tablet report...
I20260926 14:50:27.762583 17657 raft_consensus.cc:374] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:27.762636 17657 raft_consensus.cc:729] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3026ec8085324ea983bd05c67d7172d1, State: Initialized, Role: FOLLOWER
I20260926 14:50:27.763046 17657 consensus_queue.cc:260] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 43833 } } peers { permanent_uuid: "11049d7ee6344fe9986736ed99af9a93" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43429 } } peers { permanent_uuid: "3026ec8085324ea983bd05c67d7172d1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44697 } }
I20260926 14:50:27.763629 17380 heartbeater.cc:503] Master 127.12.68.254:43907 was elected leader, sending a full tablet report...
I20260926 14:50:27.764168 17656 ts_tablet_manager.cc:1428] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:50:27.764881 17657 ts_tablet_manager.cc:1428] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:50:27.764811 17658 raft_consensus.cc:348] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 43833 } } peers { permanent_uuid: "11049d7ee6344fe9986736ed99af9a93" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43429 } } peers { permanent_uuid: "3026ec8085324ea983bd05c67d7172d1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44697 } }
I20260926 14:50:27.765051 17658 raft_consensus.cc:374] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:27.765105 17658 raft_consensus.cc:729] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3a4d7beda44b2cbe1a54abe8177931, State: Initialized, Role: FOLLOWER
I20260926 14:50:27.765496 17658 consensus_queue.cc:260] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 43833 } } peers { permanent_uuid: "11049d7ee6344fe9986736ed99af9a93" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43429 } } peers { permanent_uuid: "3026ec8085324ea983bd05c67d7172d1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44697 } }
I20260926 14:50:27.766062 17642 heartbeater.cc:503] Master 127.12.68.254:43907 was elected leader, sending a full tablet report...
I20260926 14:50:27.767215 17658 ts_tablet_manager.cc:1428] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260926 14:50:27.803166 17512 tablet.cc:2369] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:27.814924 17664 raft_consensus.cc:482] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:27.815091 17664 raft_consensus.cc:504] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 43833 } } peers { permanent_uuid: "11049d7ee6344fe9986736ed99af9a93" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43429 } } peers { permanent_uuid: "3026ec8085324ea983bd05c67d7172d1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44697 } }
I20260926 14:50:27.816098 17664 leader_election.cc:288] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11049d7ee6344fe9986736ed99af9a93 (127.12.68.194:43429), 3026ec8085324ea983bd05c67d7172d1 (127.12.68.193:44697)
I20260926 14:50:27.817839 17644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.816327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44697 (local address 127.12.68.195:49367)
0926 14:50:27.816453 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:27.816469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:27.816550 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:27.816812 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:27.816815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:27.816831 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:27.817354 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:27.817356 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:27.817366 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:27.817391 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:27.817610 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:27.817613 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:27.817616 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:27.817684 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:50:27.817716 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 14:50:27.817840 17654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.816452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:49367 (local address 127.12.68.193:44697)
0926 14:50:27.816596 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.816600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.816614 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:50:27.816658 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.816661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.816698 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.816779 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.817504 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.817507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.817510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:27.817522 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.817540 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:27.817547 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.817618 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.817679 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.817716 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 14:50:27.818137 17655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.816444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:42911 (local address 127.12.68.194:43429)
0926 14:50:27.816566 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.816571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.817034 (+ 463us) server_negotiation.cc:407] Connection header received
0926 14:50:27.817086 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.817089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.817124 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.817207 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.817770 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.817773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.817776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:27.817787 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.817804 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:27.817812 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.817883 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.817987 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.818026 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:50:27.818481 17665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.816327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:43429 (local address 127.12.68.195:42911)
0926 14:50:27.816923 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:27.816936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:27.817015 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:27.817235 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:27.817238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:27.817261 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:27.817604 (+ 343us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:27.817607 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:27.817626 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:27.817654 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:27.817880 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:27.817883 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:27.817886 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:27.818319 (+ 433us) client_negotiation.cc:240] Negotiation successful
0926 14:50:27.818353 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":254,"thread_start_us":118,"threads_started":1}
I20260926 14:50:27.818419 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e83c9200bd42179f3c70435f0858cb" candidate_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3026ec8085324ea983bd05c67d7172d1" is_pre_election: true
I20260926 14:50:27.818720 17334 raft_consensus.cc:2454] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d3a4d7beda44b2cbe1a54abe8177931 in term 0.
I20260926 14:50:27.818994 17465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e83c9200bd42179f3c70435f0858cb" candidate_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11049d7ee6344fe9986736ed99af9a93" is_pre_election: true
I20260926 14:50:27.819166 17529 leader_election.cc:302] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a4d7beda44b2cbe1a54abe8177931, 3026ec8085324ea983bd05c67d7172d1; no voters:
I20260926 14:50:27.819262 17465 raft_consensus.cc:2454] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d3a4d7beda44b2cbe1a54abe8177931 in term 0.
I20260926 14:50:27.819419 17664 raft_consensus.cc:2781] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:27.819499 17664 raft_consensus.cc:482] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:27.819550 17664 raft_consensus.cc:3037] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:27.820557 17664 raft_consensus.cc:504] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 43833 } } peers { permanent_uuid: "11049d7ee6344fe9986736ed99af9a93" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43429 } } peers { permanent_uuid: "3026ec8085324ea983bd05c67d7172d1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44697 } }
I20260926 14:50:27.820853 17664 leader_election.cc:288] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [CANDIDATE]: Term 1 election: Requested vote from peers 11049d7ee6344fe9986736ed99af9a93 (127.12.68.194:43429), 3026ec8085324ea983bd05c67d7172d1 (127.12.68.193:44697)
I20260926 14:50:27.821131 17465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e83c9200bd42179f3c70435f0858cb" candidate_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11049d7ee6344fe9986736ed99af9a93"
I20260926 14:50:27.821171 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e83c9200bd42179f3c70435f0858cb" candidate_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3026ec8085324ea983bd05c67d7172d1"
I20260926 14:50:27.821245 17465 raft_consensus.cc:3037] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:27.821264 17334 raft_consensus.cc:3037] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:27.822425 17334 raft_consensus.cc:2454] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a4d7beda44b2cbe1a54abe8177931 in term 1.
I20260926 14:50:27.822425 17465 raft_consensus.cc:2454] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a4d7beda44b2cbe1a54abe8177931 in term 1.
I20260926 14:50:27.822677 17529 leader_election.cc:302] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a4d7beda44b2cbe1a54abe8177931, 3026ec8085324ea983bd05c67d7172d1; no voters:
I20260926 14:50:27.823204 17664 raft_consensus.cc:2781] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:27.823395 17664 raft_consensus.cc:686] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [term 1 LEADER]: Becoming Leader. State: Replica: 2d3a4d7beda44b2cbe1a54abe8177931, State: Running, Role: LEADER
I20260926 14:50:27.823696 17664 consensus_queue.cc:237] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 43833 } } peers { permanent_uuid: "11049d7ee6344fe9986736ed99af9a93" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43429 } } peers { permanent_uuid: "3026ec8085324ea983bd05c67d7172d1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44697 } }
I20260926 14:50:27.826632 17191 catalog_manager.cc:5495] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d3a4d7beda44b2cbe1a54abe8177931 (127.12.68.195). New cstate: current_term: 1 leader_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a4d7beda44b2cbe1a54abe8177931" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 43833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11049d7ee6344fe9986736ed99af9a93" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3026ec8085324ea983bd05c67d7172d1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44697 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:27.838721 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.837380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.12.68.195:43833)
0926 14:50:27.837494 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.837498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.837511 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:27.837659 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.837662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.837715 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.837780 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.838390 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.838393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.838396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:27.838407 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.838425 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:27.838434 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.838503 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.838572 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.838609 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:50:27.844038 17465 raft_consensus.cc:1261] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93 [term 1 FOLLOWER]: Refusing update from remote peer 2d3a4d7beda44b2cbe1a54abe8177931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:27.844039 17334 raft_consensus.cc:1261] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1 [term 1 FOLLOWER]: Refusing update from remote peer 2d3a4d7beda44b2cbe1a54abe8177931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:27.844610 17666 consensus_queue.cc:1035] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3026ec8085324ea983bd05c67d7172d1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:27.844767 17664 consensus_queue.cc:1035] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11049d7ee6344fe9986736ed99af9a93" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 43429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:27.859306 17192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:27.859444 17192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:27.861546 17192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35d3a627f7f44a17b8870fb452183909: Sending DeleteTablet for 3 replicas of tablet c1e83c9200bd42179f3c70435f0858cb
I20260926 14:50:27.862277 17576 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e83c9200bd42179f3c70435f0858cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:27 UTC) from {username='slave'} at 127.0.0.1:34076
I20260926 14:50:27.862279 17445 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e83c9200bd42179f3c70435f0858cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:27 UTC) from {username='slave'} at 127.0.0.1:52988
I20260926 14:50:27.862303 17314 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e83c9200bd42179f3c70435f0858cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:27 UTC) from {username='slave'} at 127.0.0.1:49498
I20260926 14:50:27.862701 17679 tablet_replica.cc:330] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93: stopping tablet replica
I20260926 14:50:27.862702 17680 tablet_replica.cc:330] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931: stopping tablet replica
I20260926 14:50:27.862921 17681 tablet_replica.cc:330] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1: stopping tablet replica
I20260926 14:50:27.862984 17680 raft_consensus.cc:2229] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:27.863152 17679 raft_consensus.cc:2229] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:27.863152 17681 raft_consensus.cc:2229] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:27.863214 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 17252
I20260926 14:50:27.863292 17680 raft_consensus.cc:2258] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:27.863413 17681 raft_consensus.cc:2258] T c1e83c9200bd42179f3c70435f0858cb P 3026ec8085324ea983bd05c67d7172d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:27.864795 17679 raft_consensus.cc:2258] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:27.865955 17679 ts_tablet_manager.cc:1905] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:27.868201 17679 ts_tablet_manager.cc:1918] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:27.868289 17679 log.cc:1196] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/c1e83c9200bd42179f3c70435f0858cb
I20260926 14:50:27.868577 17679 ts_tablet_manager.cc:1939] T c1e83c9200bd42179f3c70435f0858cb P 11049d7ee6344fe9986736ed99af9a93: Deleting consensus metadata
I20260926 14:50:27.869333 17680 ts_tablet_manager.cc:1905] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:27.870046 17177 catalog_manager.cc:4841] TS 11049d7ee6344fe9986736ed99af9a93 (127.12.68.194:43429): tablet c1e83c9200bd42179f3c70435f0858cb (table test-table [id=d7e74a0e842d4b529cf12f8fd1958247]) successfully deleted
W20260926 14:50:27.870467 17176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:44697 (error 108)
I20260926 14:50:27.870666 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 17383
I20260926 14:50:27.872198 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.871846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44697 (local address 127.0.0.1:49510)
0926 14:50:27.872047 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:27.872118 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260926 14:50:27.872370 17176 catalog_manager.cc:4584] TS 3026ec8085324ea983bd05c67d7172d1 (127.12.68.193:44697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c1e83c9200bd42179f3c70435f0858cb: Network error: Client connection negotiation failed: client connection to 127.12.68.193:44697: connect: Connection refused (error 111)
I20260926 14:50:27.872725 17680 ts_tablet_manager.cc:1918] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:27.872818 17680 log.cc:1196] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/c1e83c9200bd42179f3c70435f0858cb
I20260926 14:50:27.873081 17680 ts_tablet_manager.cc:1939] T c1e83c9200bd42179f3c70435f0858cb P 2d3a4d7beda44b2cbe1a54abe8177931: Deleting consensus metadata
I20260926 14:50:27.873945 17177 catalog_manager.cc:4841] TS 2d3a4d7beda44b2cbe1a54abe8177931 (127.12.68.195:43833): tablet c1e83c9200bd42179f3c70435f0858cb (table test-table [id=d7e74a0e842d4b529cf12f8fd1958247]) successfully deleted
I20260926 14:50:27.876972 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 17514
I20260926 14:50:27.881438 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 17161
2026-09-26T14:50:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (883 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 14:50:27.900926 12563 mini_oidc.cc:152] Starting JWKS server
I20260926 14:50:27.902402 12563 webserver.cc:460] Webserver started at https://127.0.0.1:37723/ using document root <none> and password file <none>
I20260926 14:50:27.902554 12563 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 14:50:27.902844 12563 webserver.cc:460] Webserver started at http://127.0.0.1:46535/ using document root <none> and password file <none>
2026-09-26T14:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:27Z Disabled control of system clock
I20260926 14:50:27.910531 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:43061
--webserver_interface=127.12.68.254
--webserver_port=0
--builtin_ntp_servers=127.12.68.212:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:43061
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37723/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/testchainca.cert with env {}
W20260926 14:50:28.012770 17688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:28.013010 17688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:28.013046 17688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:28.016927 17688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:28.016973 17688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:28.016986 17688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:28.017004 17688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:28.017031 17688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:50:28.017061 17688 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:50:28.017084 17688 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37723/jwks.json
I20260926 14:50:28.022459 17688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:43061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:43061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37723/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:28.023391 17688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:28.024333 17688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:28.031991 17696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:28.032084 17688 server_base.cc:1034] running on GCE node
W20260926 14:50:28.032006 17694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.032013 17693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:28.032862 17688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:28.033825 17688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:28.034972 17688 hybrid_clock.cc:648] HybridClock initialized: now 1790434228034957 us; error 27 us; skew 500 ppm
I20260926 14:50:28.036626 17688 webserver.cc:460] Webserver started at http://127.12.68.254:42807/ using document root <none> and password file <none>
I20260926 14:50:28.037130 17688 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:28.037192 17688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:28.037359 17688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:28.039198 17688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "eff7e676fcc1469e9cddded28a41bd5e"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-24mn"
server_key: "6d84351675ca519a7b7dcc625e995fb5"
server_key_iv: "36738a0d50d9593b128f110e86052065"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.039709 17688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "eff7e676fcc1469e9cddded28a41bd5e"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-24mn"
server_key: "6d84351675ca519a7b7dcc625e995fb5"
server_key_iv: "36738a0d50d9593b128f110e86052065"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.042970 17688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 14:50:28.045100 17702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.046187 17688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:28.046289 17688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "eff7e676fcc1469e9cddded28a41bd5e"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-24mn"
server_key: "6d84351675ca519a7b7dcc625e995fb5"
server_key_iv: "36738a0d50d9593b128f110e86052065"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.046373 17688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:28.066072 17688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:28.066633 17688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:28.066793 17688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:28.073148 17688 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:43061
I20260926 14:50:28.073165 17754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:43061 every 8 connection(s)
I20260926 14:50:28.073974 17688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:28.074939 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 17688
I20260926 14:50:28.075034 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:28.075284 12563 external_mini_cluster.cc:1428] Setting key 47ae1f3c5fe07bb05157e64874b3759f
I20260926 14:50:28.077255 17755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:28.082232 17755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e: Bootstrap starting.
I20260926 14:50:28.083566 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.076485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.12.68.254:43061)
0926 14:50:28.077088 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.077101 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.077134 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:50:28.077805 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.077822 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.078189 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.078488 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.078886 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.079812 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.080692 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.081132 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.081276 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.082002 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.082007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.082019 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:28.082391 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.082448 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:28.082469 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.082550 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.082681 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.082918 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":137,"threads_started":1}
I20260926 14:50:28.084300 17755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:28.084945 17755 log.cc:824] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:28.086045 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:50:28.086639 17755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e: No bootstrap required, opened a new log
I20260926 14:50:28.089859 17755 raft_consensus.cc:348] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff7e676fcc1469e9cddded28a41bd5e" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43061 } }
I20260926 14:50:28.090049 17755 raft_consensus.cc:374] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:28.090101 17755 raft_consensus.cc:729] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff7e676fcc1469e9cddded28a41bd5e, State: Initialized, Role: FOLLOWER
I20260926 14:50:28.090471 17755 consensus_queue.cc:260] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff7e676fcc1469e9cddded28a41bd5e" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43061 } }
I20260926 14:50:28.090583 17755 raft_consensus.cc:388] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:28.090643 17755 raft_consensus.cc:482] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:28.090741 17755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:28.091464 17755 raft_consensus.cc:504] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff7e676fcc1469e9cddded28a41bd5e" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43061 } }
I20260926 14:50:28.091756 17755 leader_election.cc:302] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eff7e676fcc1469e9cddded28a41bd5e; no voters:
I20260926 14:50:28.092017 17755 leader_election.cc:288] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:28.092105 17760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:28.092311 17760 raft_consensus.cc:686] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [term 1 LEADER]: Becoming Leader. State: Replica: eff7e676fcc1469e9cddded28a41bd5e, State: Running, Role: LEADER
I20260926 14:50:28.092586 17760 consensus_queue.cc:237] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff7e676fcc1469e9cddded28a41bd5e" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43061 } }
I20260926 14:50:28.093113 17755 sys_catalog.cc:564] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:28.094206 17762 sys_catalog.cc:455] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader eff7e676fcc1469e9cddded28a41bd5e. Latest consensus state: current_term: 1 leader_uuid: "eff7e676fcc1469e9cddded28a41bd5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff7e676fcc1469e9cddded28a41bd5e" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43061 } } }
I20260926 14:50:28.094206 17761 sys_catalog.cc:455] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eff7e676fcc1469e9cddded28a41bd5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff7e676fcc1469e9cddded28a41bd5e" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43061 } } }
I20260926 14:50:28.094344 17761 sys_catalog.cc:458] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:28.094344 17762 sys_catalog.cc:458] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:28.094728 17770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:28.097237 17770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:28.102232 17770 catalog_manager.cc:1347] Generated new cluster ID: 9c8c2e9bde544e9981cf3890d30aedd2
I20260926 14:50:28.102299 17770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:28.112462 17770 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:28.113282 17770 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:28.116552 17770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eff7e676fcc1469e9cddded28a41bd5e: Generated new TSK 0
I20260926 14:50:28.117110 17770 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 14:50:28.149133 17785 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 14:50:28.149251 17785 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 14:50:28.149700 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.133649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.12.68.254:43061)
0926 14:50:28.133761 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.133766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.133940 (+ 174us) server_negotiation.cc:407] Connection header received
0926 14:50:28.134175 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.134178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.134223 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.134316 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:50:28.134581 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.135119 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.136349 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.136572 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.136655 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.136974 (+ 319us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:50:28.149371 (+ 12397us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 14:50:28.149599 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33292: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 14:50:28.149875 17758 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33292: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 14:50:28.151080 17784 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:28.133546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43061 (local address 127.0.0.1:33292)
0926 14:50:28.133862 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.133876 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.133996 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.134338 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.134341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.134348 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:50:28.134474 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.134480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.135215 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.135219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.136218 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.136228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.136675 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.136678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.136772 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.136821 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:50:28.149588 (+ 12767us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 14:50:28.149803 (+ 215us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.68.254:43061: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":107,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.68.254:43061: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 14:50:28.151872 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 17688
2026-09-26T14:50:28Z chronyd exiting
I20260926 14:50:28.187973 12563 test_util.cc:175] -----------------------------------------------
I20260926 14:50:28.188047 12563 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434204424021-12563-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (288 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 14:50:28.188999 12563 mini_oidc.cc:152] Starting JWKS server
I20260926 14:50:28.189790 12563 webserver.cc:460] Webserver started at https://127.0.0.1:40113/ using document root <none> and password file <none>
I20260926 14:50:28.189867 12563 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 14:50:28.190064 12563 webserver.cc:460] Webserver started at http://127.0.0.1:41781/ using document root <none> and password file <none>
2026-09-26T14:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:28Z Disabled control of system clock
I20260926 14:50:28.198132 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:40969
--webserver_interface=127.12.68.254
--webserver_port=0
--builtin_ntp_servers=127.12.68.212:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:40969
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40113/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/testchainca.cert with env {}
W20260926 14:50:28.300714 17791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:28.300951 17791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:28.300987 17791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:28.304976 17791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:28.305023 17791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:28.305037 17791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:28.305051 17791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:28.305065 17791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:50:28.305085 17791 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:50:28.305097 17791 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40113/jwks.json
I20260926 14:50:28.310097 17791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:40969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40113/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:28.310930 17791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:28.311769 17791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:28.319418 17799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.319424 17796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.319424 17797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:28.319591 17791 server_base.cc:1034] running on GCE node
I20260926 14:50:28.320286 17791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:28.321267 17791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:28.322430 17791 hybrid_clock.cc:648] HybridClock initialized: now 1790434228322415 us; error 25 us; skew 500 ppm
I20260926 14:50:28.324062 17791 webserver.cc:460] Webserver started at http://127.12.68.254:41991/ using document root <none> and password file <none>
I20260926 14:50:28.324559 17791 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:28.324615 17791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:28.324812 17791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:28.326661 17791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "e245c2ebae0b45bab777f0cacada1366"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-24mn"
server_key: "900d3729b943cb6b34cc44f324d07e04"
server_key_iv: "c855a001cf0bc28e8f37e574bcab26c8"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.327162 17791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "e245c2ebae0b45bab777f0cacada1366"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-24mn"
server_key: "900d3729b943cb6b34cc44f324d07e04"
server_key_iv: "c855a001cf0bc28e8f37e574bcab26c8"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.330471 17791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260926 14:50:28.332604 17805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.333633 17791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 14:50:28.333752 17791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "e245c2ebae0b45bab777f0cacada1366"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-24mn"
server_key: "900d3729b943cb6b34cc44f324d07e04"
server_key_iv: "c855a001cf0bc28e8f37e574bcab26c8"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.333837 17791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:28.345652 17791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:28.346192 17791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:28.346338 17791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:28.352177 17791 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:40969
I20260926 14:50:28.352182 17857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:40969 every 8 connection(s)
I20260926 14:50:28.352968 17791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:28.355743 17858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:28.360687 17858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366: Bootstrap starting.
I20260926 14:50:28.362821 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 17791
I20260926 14:50:28.362942 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:28.362934 17858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:28.363232 12563 external_mini_cluster.cc:1428] Setting key ba271d039369e1411ee66ed90efa542e
I20260926 14:50:28.363729 17858 log.cc:824] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:28.365849 17858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366: No bootstrap required, opened a new log
I20260926 14:50:28.369915 17858 raft_consensus.cc:348] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e245c2ebae0b45bab777f0cacada1366" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40969 } }
I20260926 14:50:28.370162 17858 raft_consensus.cc:374] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:28.370217 17858 raft_consensus.cc:729] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e245c2ebae0b45bab777f0cacada1366, State: Initialized, Role: FOLLOWER
I20260926 14:50:28.370657 17858 consensus_queue.cc:260] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e245c2ebae0b45bab777f0cacada1366" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40969 } }
I20260926 14:50:28.370802 17858 raft_consensus.cc:388] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:28.370877 17858 raft_consensus.cc:482] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:28.370965 17858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:28.371950 17858 raft_consensus.cc:504] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e245c2ebae0b45bab777f0cacada1366" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40969 } }
I20260926 14:50:28.372278 17858 leader_election.cc:302] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e245c2ebae0b45bab777f0cacada1366; no voters:
I20260926 14:50:28.372575 17858 leader_election.cc:288] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:28.372646 17863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:28.372763 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.364236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.12.68.254:40969)
0926 14:50:28.364865 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.364885 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.364924 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:50:28.365714 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.365735 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.366127 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.366472 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.366868 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.368335 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.369231 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.369888 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.370041 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.370809 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.370815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.370831 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:28.371294 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.371371 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:28.371396 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.371490 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.371588 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.371842 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":111,"threads_started":1}
I20260926 14:50:28.372844 17863 raft_consensus.cc:686] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [term 1 LEADER]: Becoming Leader. State: Replica: e245c2ebae0b45bab777f0cacada1366, State: Running, Role: LEADER
I20260926 14:50:28.373124 17863 consensus_queue.cc:237] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e245c2ebae0b45bab777f0cacada1366" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40969 } }
I20260926 14:50:28.373708 17858 sys_catalog.cc:564] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:28.374178 17863 sys_catalog.cc:455] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e245c2ebae0b45bab777f0cacada1366. Latest consensus state: current_term: 1 leader_uuid: "e245c2ebae0b45bab777f0cacada1366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e245c2ebae0b45bab777f0cacada1366" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40969 } } }
I20260926 14:50:28.374310 17863 sys_catalog.cc:458] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:28.374178 17864 sys_catalog.cc:455] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e245c2ebae0b45bab777f0cacada1366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e245c2ebae0b45bab777f0cacada1366" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40969 } } }
I20260926 14:50:28.374387 17864 sys_catalog.cc:458] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:28.375605 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:50:28.375774 17868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:28.378633 17868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:28.383366 17868 catalog_manager.cc:1347] Generated new cluster ID: 4bad15ccbb764fc2aa3a48b8fc7140da
I20260926 14:50:28.383425 17868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:28.399003 17868 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:28.399784 17868 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:28.405859 17868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e245c2ebae0b45bab777f0cacada1366: Generated new TSK 0
I20260926 14:50:28.406402 17868 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 14:50:28.430245 17888 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 14:50:28.430300 17888 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 14:50:28.431042 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.414781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.12.68.254:40969)
0926 14:50:28.414878 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.414882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.415078 (+ 196us) server_negotiation.cc:407] Connection header received
0926 14:50:28.415216 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.415219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.415269 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.415363 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:50:28.415588 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.416164 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.417056 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.417423 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.417514 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.417860 (+ 346us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:50:28.430653 (+ 12793us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40113/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 14:50:28.430920 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52156: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40113/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 14:50:28.431100 17862 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52156: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40113/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 14:50:28.431118 17887 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:28.414737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40969 (local address 127.0.0.1:52156)
0926 14:50:28.415002 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.415014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.415112 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.415375 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.415379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.415387 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:50:28.415504 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.415510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.416265 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.416269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.416899 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.416909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.417555 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.417558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.417642 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.417669 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:50:28.430852 (+ 13183us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40113/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 14:50:28.430989 (+ 137us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.68.254:40969: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40113/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":105,"threads_started":1}
I20260926 14:50:28.431638 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 17791
2026-09-26T14:50:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (280 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 14:50:28.469430 12563 mini_oidc.cc:152] Starting JWKS server
I20260926 14:50:28.470250 12563 webserver.cc:460] Webserver started at https://127.0.0.1:41011/ using document root <none> and password file <none>
I20260926 14:50:28.470342 12563 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 14:50:28.470574 12563 webserver.cc:460] Webserver started at http://127.0.0.1:34073/ using document root <none> and password file <none>
2026-09-26T14:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:28Z Disabled control of system clock
I20260926 14:50:28.478384 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:37893
--webserver_interface=127.12.68.254
--webserver_port=0
--builtin_ntp_servers=127.12.68.212:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:37893
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41011/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 14:50:28.581511 17894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:28.581729 17894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:28.581771 17894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:28.585608 17894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:28.585655 17894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:28.585669 17894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:28.585685 17894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:28.585698 17894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:50:28.585711 17894 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:50:28.585727 17894 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41011/jwks.json
I20260926 14:50:28.590219 17894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:37893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:37893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41011/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:28.591080 17894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:28.592337 17894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:28.599749 17902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.599766 17900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.599798 17899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:28.599915 17894 server_base.cc:1034] running on GCE node
I20260926 14:50:28.600513 17894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:28.601446 17894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:28.602592 17894 hybrid_clock.cc:648] HybridClock initialized: now 1790434228602585 us; error 34 us; skew 500 ppm
I20260926 14:50:28.604418 17894 webserver.cc:460] Webserver started at http://127.12.68.254:37141/ using document root <none> and password file <none>
I20260926 14:50:28.604955 17894 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:28.605021 17894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:28.605201 17894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:28.607029 17894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "3a35e9b3efdf42f5b638f00d4d4fe331"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-24mn"
server_key: "6cdf57c15614705666efa3f738dc8e89"
server_key_iv: "824cf8421404134847f8eb97167d55d2"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.607517 17894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "3a35e9b3efdf42f5b638f00d4d4fe331"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-24mn"
server_key: "6cdf57c15614705666efa3f738dc8e89"
server_key_iv: "824cf8421404134847f8eb97167d55d2"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.610816 17894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:50:28.612924 17908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.613979 17894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:28.614089 17894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "3a35e9b3efdf42f5b638f00d4d4fe331"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-24mn"
server_key: "6cdf57c15614705666efa3f738dc8e89"
server_key_iv: "824cf8421404134847f8eb97167d55d2"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.614172 17894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:28.627022 17894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:28.627555 17894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:28.627697 17894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:28.633682 17894 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:37893
I20260926 14:50:28.633703 17960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:37893 every 8 connection(s)
I20260926 14:50:28.634454 17894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:28.637202 17961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:28.642007 17961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331: Bootstrap starting.
I20260926 14:50:28.642774 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 17894
I20260926 14:50:28.642887 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:28.643164 12563 external_mini_cluster.cc:1428] Setting key 46f57deb7c3e5a7c4cc589dd12f6a4a3
I20260926 14:50:28.644579 17961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:28.645223 17961 log.cc:824] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:28.646920 17961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331: No bootstrap required, opened a new log
I20260926 14:50:28.650686 17961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35e9b3efdf42f5b638f00d4d4fe331" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37893 } }
I20260926 14:50:28.650887 17961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:28.650926 17961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a35e9b3efdf42f5b638f00d4d4fe331, State: Initialized, Role: FOLLOWER
I20260926 14:50:28.651281 17964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.644109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.12.68.254:37893)
0926 14:50:28.644648 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.644665 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.644700 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:50:28.645274 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.645294 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.645660 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.645920 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.646215 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.647156 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.648337 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.648796 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.648910 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.649655 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.649660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.649671 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:28.650018 (+ 347us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.650069 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:28.650088 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.650157 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.650293 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.650537 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":103,"threads_started":1}
I20260926 14:50:28.651326 17961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35e9b3efdf42f5b638f00d4d4fe331" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37893 } }
I20260926 14:50:28.651456 17961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:28.651516 17961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:28.651608 17961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:28.652388 17961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35e9b3efdf42f5b638f00d4d4fe331" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37893 } }
I20260926 14:50:28.652676 17961 leader_election.cc:302] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a35e9b3efdf42f5b638f00d4d4fe331; no voters:
I20260926 14:50:28.652943 17961 leader_election.cc:288] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:28.653008 17966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:28.653173 17966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [term 1 LEADER]: Becoming Leader. State: Replica: 3a35e9b3efdf42f5b638f00d4d4fe331, State: Running, Role: LEADER
I20260926 14:50:28.653436 17966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35e9b3efdf42f5b638f00d4d4fe331" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37893 } }
I20260926 14:50:28.653738 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:50:28.654059 17961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:28.654927 17968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a35e9b3efdf42f5b638f00d4d4fe331. Latest consensus state: current_term: 1 leader_uuid: "3a35e9b3efdf42f5b638f00d4d4fe331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35e9b3efdf42f5b638f00d4d4fe331" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37893 } } }
I20260926 14:50:28.655035 17968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:28.655372 17975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:28.655335 17967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a35e9b3efdf42f5b638f00d4d4fe331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35e9b3efdf42f5b638f00d4d4fe331" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37893 } } }
I20260926 14:50:28.655428 17967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:28.658242 17975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:28.662740 17975 catalog_manager.cc:1347] Generated new cluster ID: 1b7e63a80e1e419987e92b6192ff972c
I20260926 14:50:28.662801 17975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:28.700839 17975 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:28.701727 17975 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:28.711382 17975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a35e9b3efdf42f5b638f00d4d4fe331: Generated new TSK 0
I20260926 14:50:28.712036 17975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:50:28.728044 17964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.723331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.12.68.254:37893)
0926 14:50:28.723457 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.723462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.723625 (+ 163us) server_negotiation.cc:407] Connection header received
0926 14:50:28.723775 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.723777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.723829 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.723949 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:50:28.724174 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.724684 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.725866 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.726066 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.726132 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.726420 (+ 288us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:50:28.727643 (+ 1223us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41011/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 14:50:28.727867 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33428: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41011/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 14:50:28.728103 17964 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33428: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41011/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 14:50:28.729918 17990 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:28.723232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:37893 (local address 127.0.0.1:33428)
0926 14:50:28.723535 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.723549 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.723666 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.723949 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.723952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.723962 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:50:28.724087 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.724093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.724778 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.724782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.725769 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.725779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.726161 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.726165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.726254 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.726282 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:50:28.727801 (+ 1519us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41011/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 14:50:28.729750 (+ 1949us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.68.254:37893: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41011/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":1761,"thread_start_us":94,"threads_started":1}
I20260926 14:50:28.730429 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 17894
E20260926 14:50:28.738529 17991 webserver.cc:533] Webserver: sslize failed:
E20260926 14:50:28.738590 17991 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T14:50:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (298 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T14:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:28Z Disabled control of system clock
I20260926 14:50:28.775595 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:41851
--webserver_interface=127.12.68.254
--webserver_port=0
--builtin_ntp_servers=127.12.68.212:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:41851
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/testchainca.cert with env {}
W20260926 14:50:28.879317 17995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:28.879539 17995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:28.879573 17995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:28.883423 17995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:28.883472 17995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:28.883486 17995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:28.883500 17995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:28.883514 17995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:50:28.883528 17995 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:50:28.883544 17995 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 14:50:28.888492 17995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:41851
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:41851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:28.889351 17995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:28.890190 17995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:28.897876 18003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:28.898023 17995 server_base.cc:1034] running on GCE node
W20260926 14:50:28.897862 18000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.897930 18001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:28.898620 17995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:28.899670 17995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:28.900815 17995 hybrid_clock.cc:648] HybridClock initialized: now 1790434228900790 us; error 34 us; skew 500 ppm
I20260926 14:50:28.902444 17995 webserver.cc:460] Webserver started at http://127.12.68.254:42911/ using document root <none> and password file <none>
I20260926 14:50:28.902937 17995 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:28.902994 17995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:28.903169 17995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:28.905138 17995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "f143c9b4990a48178904373b7575601e"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-24mn"
server_key: "152a2c6006327f3732d1559be1523dca"
server_key_iv: "974a92dc72548a1df50588f53b890e0b"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.905687 17995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "f143c9b4990a48178904373b7575601e"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-24mn"
server_key: "152a2c6006327f3732d1559be1523dca"
server_key_iv: "974a92dc72548a1df50588f53b890e0b"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.908988 17995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:28.911087 18009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.912194 17995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:28.912290 17995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "f143c9b4990a48178904373b7575601e"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-24mn"
server_key: "152a2c6006327f3732d1559be1523dca"
server_key_iv: "974a92dc72548a1df50588f53b890e0b"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.912386 17995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:28.926302 17995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:28.926842 17995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:28.926991 17995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:28.933153 17995 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:41851
I20260926 14:50:28.933163 18061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:41851 every 8 connection(s)
I20260926 14:50:28.933952 17995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:28.936630 18062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:28.939996 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 17995
I20260926 14:50:28.940109 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:28.940389 12563 external_mini_cluster.cc:1428] Setting key 3f00064a2c18551d18fb7fb1cb7817e0
I20260926 14:50:28.942040 18062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e: Bootstrap starting.
I20260926 14:50:28.944058 18062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:28.944706 18062 log.cc:824] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:28.946520 18062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e: No bootstrap required, opened a new log
I20260926 14:50:28.948935 18065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.941600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.12.68.254:41851)
0926 14:50:28.942217 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.942231 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.942264 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:50:28.942865 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.942880 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.943250 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.943544 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.943976 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.944956 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.946070 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.946506 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.946643 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.947384 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.947389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.947400 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:28.947740 (+ 340us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.947789 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:28.947806 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.947908 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.948039 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.948277 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":130,"threads_started":1}
I20260926 14:50:28.950073 18062 raft_consensus.cc:348] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f143c9b4990a48178904373b7575601e" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41851 } }
I20260926 14:50:28.950314 18062 raft_consensus.cc:374] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:28.950366 18062 raft_consensus.cc:729] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f143c9b4990a48178904373b7575601e, State: Initialized, Role: FOLLOWER
I20260926 14:50:28.950742 18062 consensus_queue.cc:260] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f143c9b4990a48178904373b7575601e" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41851 } }
I20260926 14:50:28.950858 18062 raft_consensus.cc:388] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:28.950914 18062 raft_consensus.cc:482] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:28.950979 18062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:28.951454 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:50:28.951795 18062 raft_consensus.cc:504] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f143c9b4990a48178904373b7575601e" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41851 } }
I20260926 14:50:28.952099 18062 leader_election.cc:302] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f143c9b4990a48178904373b7575601e; no voters:
I20260926 14:50:28.952348 18062 leader_election.cc:288] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:28.952430 18067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:28.952585 18067 raft_consensus.cc:686] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [term 1 LEADER]: Becoming Leader. State: Replica: f143c9b4990a48178904373b7575601e, State: Running, Role: LEADER
I20260926 14:50:28.952848 18067 consensus_queue.cc:237] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f143c9b4990a48178904373b7575601e" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41851 } }
I20260926 14:50:28.953532 18062 sys_catalog.cc:564] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:28.953976 18069 sys_catalog.cc:455] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f143c9b4990a48178904373b7575601e. Latest consensus state: current_term: 1 leader_uuid: "f143c9b4990a48178904373b7575601e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f143c9b4990a48178904373b7575601e" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41851 } } }
I20260926 14:50:28.954118 18069 sys_catalog.cc:458] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:28.954391 18068 sys_catalog.cc:455] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f143c9b4990a48178904373b7575601e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f143c9b4990a48178904373b7575601e" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41851 } } }
I20260926 14:50:28.954487 18068 sys_catalog.cc:458] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:28.955559 18073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:28.958156 18073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:28.962801 18073 catalog_manager.cc:1347] Generated new cluster ID: d4ee3942e30f402682de15aa5f509597
I20260926 14:50:28.962859 18073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:28.972049 18073 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:28.972783 18073 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:28.983565 18073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f143c9b4990a48178904373b7575601e: Generated new TSK 0
I20260926 14:50:28.984153 18073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:50:29.078503 18065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.993292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.12.68.254:41851)
0926 14:50:28.993413 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.993418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.993568 (+ 150us) server_negotiation.cc:407] Connection header received
0926 14:50:28.993713 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.993716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.993757 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.993837 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:50:28.994058 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.994567 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.995750 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.995997 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.996068 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.996366 (+ 298us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:50:29.078114 (+ 81748us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 14:50:29.078371 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44568: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 14:50:29.078583 18065 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44568: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 14:50:29.078745 18091 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:28.993209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:41851 (local address 127.0.0.1:44568)
0926 14:50:28.993485 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.993498 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.993606 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.993851 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.993854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.993861 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:50:28.993970 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.993976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.994658 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.994662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.995646 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.995655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.996097 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.996102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.996198 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.996225 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:50:29.078349 (+ 82124us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 14:50:29.078559 (+ 210us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.68.254:41851: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
I20260926 14:50:29.079357 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 17995
2026-09-26T14:50:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (328 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (114 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (114 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:29 dist-test-slave-24mn krb5kdc[18103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:29 dist-test-slave-24mn krb5kdc[18103](info): set up 2 sockets
Sep 26 14:50:29 dist-test-slave-24mn krb5kdc[18103](info): commencing operation
krb5kdc: starting...
W20260926 14:50:30.255934 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:50:30 dist-test-slave-24mn krb5kdc[18103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:50:30.364758 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:38073
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:38073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:30.469475 18116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:30.469713 18116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:30.469765 18116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:30.473644 18116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:30.473694 18116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:30.473721 18116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:30.473742 18116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:30.473770 18116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:30.478775 18116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:38073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:38073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:30.479727 18116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:30.480701 18116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:30.488336 18122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:30.488345 18121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:30.488453 18116 server_base.cc:1034] running on GCE node
W20260926 14:50:30.488685 18124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:30.489225 18116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:30.490233 18116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:30.491384 18116 hybrid_clock.cc:648] HybridClock initialized: now 1790434230491369 us; error 27 us; skew 500 ppm
Sep 26 14:50:30 dist-test-slave-24mn krb5kdc[18103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434230, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:30.493916 18116 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:50:30.494724 18116 webserver.cc:460] Webserver started at http://127.12.68.254:33311/ using document root <none> and password file <none>
I20260926 14:50:30.495210 18116 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:30.495268 18116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:30.495456 18116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:30.497370 18116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "ca1e8ca7ed2c4d1dbe512f8a94c5788a"
format_stamp: "Formatted at 2026-09-26 14:50:30 on dist-test-slave-24mn"
server_key: "a01b167f4eab6fbdfeeee4d2bb7b002a"
server_key_iv: "c3a4f8890bb5db90dbcb93474f3935ad"
server_key_version: "encryptionkey@0"
I20260926 14:50:30.497958 18116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "ca1e8ca7ed2c4d1dbe512f8a94c5788a"
format_stamp: "Formatted at 2026-09-26 14:50:30 on dist-test-slave-24mn"
server_key: "a01b167f4eab6fbdfeeee4d2bb7b002a"
server_key_iv: "c3a4f8890bb5db90dbcb93474f3935ad"
server_key_version: "encryptionkey@0"
I20260926 14:50:30.501350 18116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:50:30.503469 18131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:30.504595 18116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:30.504696 18116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "ca1e8ca7ed2c4d1dbe512f8a94c5788a"
format_stamp: "Formatted at 2026-09-26 14:50:30 on dist-test-slave-24mn"
server_key: "a01b167f4eab6fbdfeeee4d2bb7b002a"
server_key_iv: "c3a4f8890bb5db90dbcb93474f3935ad"
server_key_version: "encryptionkey@0"
I20260926 14:50:30.504782 18116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:30.528709 18116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:30.531584 18116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:30.531754 18116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:30.538007 18116 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:38073
I20260926 14:50:30.538064 18183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:38073 every 8 connection(s)
I20260926 14:50:30.538877 18116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:30.539546 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 18116
I20260926 14:50:30.539671 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:30.539980 12563 external_mini_cluster.cc:1428] Setting key 8a313c5564814597d4c4cef891512a00
I20260926 14:50:30.541849 18184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:30.547809 18184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a: Bootstrap starting.
Sep 26 14:50:30 dist-test-slave-24mn krb5kdc[18103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:30.549888 18184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:30.550549 18184 log.cc:824] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:30.552451 18184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a: No bootstrap required, opened a new log
I20260926 14:50:30.553004 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.541209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35578 (local address 127.12.68.254:38073)
0926 14:50:30.541564 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:30.541581 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:30.541614 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:50:30.542253 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:30.542271 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:30.542581 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:30.542848 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:30.543667 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.544655 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.545743 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.546180 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.546283 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:30.548851 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:30.548855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:30.548867 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:30.548894 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:30.550403 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:30.550807 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:30.550812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:30.550816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:30.550953 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:30.551219 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:30.551222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:30.551224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:30.551590 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:30.551726 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:30.552038 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 14:50:30.552306 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260926 14:50:30.555999 18184 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1e8ca7ed2c4d1dbe512f8a94c5788a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38073 } }
I20260926 14:50:30.556205 18184 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:30.556260 18184 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1e8ca7ed2c4d1dbe512f8a94c5788a, State: Initialized, Role: FOLLOWER
I20260926 14:50:30.556617 18184 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1e8ca7ed2c4d1dbe512f8a94c5788a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38073 } }
I20260926 14:50:30.556733 18184 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:30.556793 18184 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:30.556885 18184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:30.557637 18184 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1e8ca7ed2c4d1dbe512f8a94c5788a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38073 } }
I20260926 14:50:30.557924 18184 leader_election.cc:302] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca1e8ca7ed2c4d1dbe512f8a94c5788a; no voters:
I20260926 14:50:30.558203 18184 leader_election.cc:288] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:30.558307 18189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:30.558523 18189 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [term 1 LEADER]: Becoming Leader. State: Replica: ca1e8ca7ed2c4d1dbe512f8a94c5788a, State: Running, Role: LEADER
I20260926 14:50:30.558838 18189 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1e8ca7ed2c4d1dbe512f8a94c5788a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38073 } }
I20260926 14:50:30.559197 18184 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:30.560499 18190 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca1e8ca7ed2c4d1dbe512f8a94c5788a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1e8ca7ed2c4d1dbe512f8a94c5788a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38073 } } }
I20260926 14:50:30.560613 18190 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:30.560886 18191 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca1e8ca7ed2c4d1dbe512f8a94c5788a. Latest consensus state: current_term: 1 leader_uuid: "ca1e8ca7ed2c4d1dbe512f8a94c5788a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1e8ca7ed2c4d1dbe512f8a94c5788a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38073 } } }
I20260926 14:50:30.560971 18191 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:30.561376 18200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:30.564098 18200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:30.568988 18200 catalog_manager.cc:1347] Generated new cluster ID: 03ebe0d938854218a0c43b7e9198cbf8
I20260926 14:50:30.569053 18200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:30.581399 18200 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:30.582180 18200 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:30.587321 18200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca1e8ca7ed2c4d1dbe512f8a94c5788a: Generated new TSK 0
I20260926 14:50:30.587864 18200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:50:30.612286 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:38073
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:30.714931 18210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:30.715159 18210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:30.715199 18210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:30.719129 18210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:30.719179 18210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:30.719233 18210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:30.724090 18210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:30.724938 18210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:30.725818 18210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:30.734719 18218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:30.734710 18215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:30.734730 18216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:30.734774 18210 server_base.cc:1034] running on GCE node
I20260926 14:50:30.735234 18210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:30.735795 18210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:30.736946 18210 hybrid_clock.cc:648] HybridClock initialized: now 1790434230736924 us; error 33 us; skew 500 ppm
Sep 26 14:50:30 dist-test-slave-24mn krb5kdc[18103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434230, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:30.739430 18210 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:50:30.740245 18210 webserver.cc:460] Webserver started at http://127.12.68.193:39555/ using document root <none> and password file <none>
I20260926 14:50:30.740741 18210 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:30.740796 18210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:30.740980 18210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:30.742831 18210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "9e17a4f763bf4d62b3220f85b9393b83"
format_stamp: "Formatted at 2026-09-26 14:50:30 on dist-test-slave-24mn"
server_key: "1cc9771d74555bdd6d436501f57fc753"
server_key_iv: "969a2cd37ef797d263c1b7470d85dab6"
server_key_version: "encryptionkey@0"
I20260926 14:50:30.743314 18210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "9e17a4f763bf4d62b3220f85b9393b83"
format_stamp: "Formatted at 2026-09-26 14:50:30 on dist-test-slave-24mn"
server_key: "1cc9771d74555bdd6d436501f57fc753"
server_key_iv: "969a2cd37ef797d263c1b7470d85dab6"
server_key_version: "encryptionkey@0"
I20260926 14:50:30.746673 18210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:50:30.748870 18225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:30.749962 18210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:30.750063 18210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "9e17a4f763bf4d62b3220f85b9393b83"
format_stamp: "Formatted at 2026-09-26 14:50:30 on dist-test-slave-24mn"
server_key: "1cc9771d74555bdd6d436501f57fc753"
server_key_iv: "969a2cd37ef797d263c1b7470d85dab6"
server_key_version: "encryptionkey@0"
I20260926 14:50:30.750152 18210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:30.776062 18210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:30.778752 18210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:30.778915 18210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:30.779469 18210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:30.780298 18210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:30.780344 18210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:30.780395 18210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:30.780422 18210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:30.788611 18210 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:36769
I20260926 14:50:30.788623 18338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:36769 every 8 connection(s)
I20260926 14:50:30.789413 18210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:30.797322 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 18210
I20260926 14:50:30.797423 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:30.797691 12563 external_mini_cluster.cc:1428] Setting key 36e35d375e7f71f747694f2bdf55ed79
Sep 26 14:50:30 dist-test-slave-24mn krb5kdc[18103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434230, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:30.803532 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.791090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:45017 (local address 127.12.68.254:38073)
0926 14:50:30.791224 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:30.791227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:30.791942 (+ 715us) server_negotiation.cc:407] Connection header received
0926 14:50:30.793272 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:30.793276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:30.793334 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:30.793419 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:30.794817 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.795598 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.797095 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.797437 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.797536 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:30.800270 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:30.800273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:30.800276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:30.800302 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:30.801679 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:30.802132 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:30.802136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:30.802138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:30.802213 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:30.802515 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:30.802518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:30.802520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:30.802696 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:30.802776 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:30.803251 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 14:50:30.803379 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:50:30.804309 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.791375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:38073 (local address 127.12.68.193:45017)
0926 14:50:30.791792 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.791832 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:30.793063 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:30.793615 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:30.793624 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:30.794060 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:30.794650 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:30.794668 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.795743 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.795748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:30.796975 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:30.796987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.797695 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.797700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:30.797890 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:30.798672 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:30.798691 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:30.800121 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:30.801780 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:30.801785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:30.801800 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:30.802032 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:30.802288 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:30.802291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:30.802293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:30.802422 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:30.802788 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:30.802794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:30.803015 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 14:50:30.803171 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 14:50:30.803468 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":97,"threads_started":1}
I20260926 14:50:30.805507 18339 heartbeater.cc:347] Connected to a master server at 127.12.68.254:38073
I20260926 14:50:30.805758 18339 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:30.806308 18339 heartbeater.cc:511] Master 127.12.68.254:38073 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
I20260926 14:50:30.807730 18148 ts_manager.cc:194] Registered new tserver with Master: 9e17a4f763bf4d62b3220f85b9393b83 (127.12.68.193:36769)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
I20260926 14:50:30.809135 18148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:45017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:50:30.822552 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:38073
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:30.924748 18344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:30.924970 18344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:30.925009 18344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:30.928968 18344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:30.929016 18344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:30.929070 18344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:30.934165 18344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:30.934993 18344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:30.935853 18344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:30.944366 18349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:30.944379 18352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:30.944367 18350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:30.944612 18344 server_base.cc:1034] running on GCE node
I20260926 14:50:30.945045 18344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:30.945602 18344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:30.946775 18344 hybrid_clock.cc:648] HybridClock initialized: now 1790434230946762 us; error 30 us; skew 500 ppm
Sep 26 14:50:30 dist-test-slave-24mn krb5kdc[18103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434230, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:30.952103 18344 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:50:30.952867 18344 webserver.cc:460] Webserver started at http://127.12.68.194:43563/ using document root <none> and password file <none>
I20260926 14:50:30.953336 18344 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:30.953388 18344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:30.953549 18344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:30.955485 18344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "bdf014826bd949cbb3d8bf5b56e62f84"
format_stamp: "Formatted at 2026-09-26 14:50:30 on dist-test-slave-24mn"
server_key: "70322dd585ea0cc111e886db4e2f642d"
server_key_iv: "b30d1b6f5174be29f398036ad1179a79"
server_key_version: "encryptionkey@0"
I20260926 14:50:30.956033 18344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "bdf014826bd949cbb3d8bf5b56e62f84"
format_stamp: "Formatted at 2026-09-26 14:50:30 on dist-test-slave-24mn"
server_key: "70322dd585ea0cc111e886db4e2f642d"
server_key_iv: "b30d1b6f5174be29f398036ad1179a79"
server_key_version: "encryptionkey@0"
I20260926 14:50:30.959399 18344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:50:30.961697 18359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:30.962800 18344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:30.962893 18344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "bdf014826bd949cbb3d8bf5b56e62f84"
format_stamp: "Formatted at 2026-09-26 14:50:30 on dist-test-slave-24mn"
server_key: "70322dd585ea0cc111e886db4e2f642d"
server_key_iv: "b30d1b6f5174be29f398036ad1179a79"
server_key_version: "encryptionkey@0"
I20260926 14:50:30.962981 18344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:30.975133 18344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:30.977562 18344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:30.977720 18344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:30.978214 18344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:30.978926 18344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:30.978968 18344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:30.979008 18344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:30.979041 18344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:30.986744 18344 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:44501
I20260926 14:50:30.986770 18472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:44501 every 8 connection(s)
I20260926 14:50:30.987507 18344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:50:30 dist-test-slave-24mn krb5kdc[18103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434230, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:30.997524 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 18344
I20260926 14:50:30.997649 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:30.997982 12563 external_mini_cluster.cc:1428] Setting key 5a1807ffafc026eb3bc2acf164054e07
I20260926 14:50:30.999807 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.989130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:53443 (local address 127.12.68.254:38073)
0926 14:50:30.989263 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:30.989267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:30.989997 (+ 730us) server_negotiation.cc:407] Connection header received
0926 14:50:30.991275 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:30.991287 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:30.991344 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:30.991414 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:30.992534 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.993041 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.994110 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.994329 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.994414 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:30.996560 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:30.996563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:30.996565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:30.996588 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:30.997805 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:30.998336 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:30.998339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:30.998340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:30.998416 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:30.998718 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:30.998721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:30.998722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:30.998945 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:30.999036 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:30.999530 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 14:50:30.999654 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":54}
I20260926 14:50:31.000522 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.989464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:38073 (local address 127.12.68.194:53443)
0926 14:50:30.989865 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.989905 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:30.991077 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:30.991584 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:30.991591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:30.991955 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:30.992394 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:30.992407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.993151 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.993154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:30.993991 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:30.994000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.994455 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.994459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:30.994613 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:30.995068 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:30.995082 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:30.996409 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:30.997930 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:30.997935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:30.997945 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:30.998242 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:30.998500 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:30.998503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:30.998505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:30.998640 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:30.999025 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:30.999033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:30.999299 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 14:50:30.999466 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 14:50:30.999747 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":102,"threads_started":1}
I20260926 14:50:31.001551 18473 heartbeater.cc:347] Connected to a master server at 127.12.68.254:38073
I20260926 14:50:31.001747 18473 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:31.002167 18473 heartbeater.cc:511] Master 127.12.68.254:38073 requested a full tablet report, sending...
I20260926 14:50:31.003090 18148 ts_manager.cc:194] Registered new tserver with Master: bdf014826bd949cbb3d8bf5b56e62f84 (127.12.68.194:44501)
I20260926 14:50:31.003787 18148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:53443
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:50:31.023061 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:38073
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:31.125234 18478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:31.125456 18478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:31.125495 18478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:31.129410 18478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:31.129456 18478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:31.129510 18478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:31.134506 18478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:31.135329 18478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:31.136214 18478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:31.144846 18483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:31.144910 18486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:31.144871 18484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:31.144881 18478 server_base.cc:1034] running on GCE node
I20260926 14:50:31.145454 18478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:31.146000 18478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:31.147146 18478 hybrid_clock.cc:648] HybridClock initialized: now 1790434231147133 us; error 24 us; skew 500 ppm
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434231, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:31.149590 18478 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:50:31.150326 18478 webserver.cc:460] Webserver started at http://127.12.68.195:43293/ using document root <none> and password file <none>
I20260926 14:50:31.150799 18478 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:31.150866 18478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:31.151036 18478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:31.152968 18478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "0a438ca4cc444902b96e44aee0f68662"
format_stamp: "Formatted at 2026-09-26 14:50:31 on dist-test-slave-24mn"
server_key: "5bc7bd6037ae7ac77607446e0379220f"
server_key_iv: "7754e2ba37de6163e376afe2911a5023"
server_key_version: "encryptionkey@0"
I20260926 14:50:31.153471 18478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "0a438ca4cc444902b96e44aee0f68662"
format_stamp: "Formatted at 2026-09-26 14:50:31 on dist-test-slave-24mn"
server_key: "5bc7bd6037ae7ac77607446e0379220f"
server_key_iv: "7754e2ba37de6163e376afe2911a5023"
server_key_version: "encryptionkey@0"
I20260926 14:50:31.156744 18478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:31.158804 18493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:31.159981 18478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:31.160069 18478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "0a438ca4cc444902b96e44aee0f68662"
format_stamp: "Formatted at 2026-09-26 14:50:31 on dist-test-slave-24mn"
server_key: "5bc7bd6037ae7ac77607446e0379220f"
server_key_iv: "7754e2ba37de6163e376afe2911a5023"
server_key_version: "encryptionkey@0"
I20260926 14:50:31.160169 18478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:31.177352 18478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:31.179688 18478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:31.179819 18478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:31.180344 18478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:31.181049 18478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:31.181093 18478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:31.181128 18478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:31.181149 18478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:31.188757 18478 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:38657
I20260926 14:50:31.188792 18606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:38657 every 8 connection(s)
I20260926 14:50:31.189515 18478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434231, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:31.197949 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 18478
I20260926 14:50:31.198046 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:31.198295 12563 external_mini_cluster.cc:1428] Setting key 71ed974a1d8450ed5c2d6e4429530825
I20260926 14:50:31.201114 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:31.191018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:57149 (local address 127.12.68.254:38073)
0926 14:50:31.191169 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:31.191173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:31.191909 (+ 736us) server_negotiation.cc:407] Connection header received
0926 14:50:31.193097 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:31.193100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:31.193145 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:31.193219 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:31.194215 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:31.194709 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:31.195694 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:31.195920 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:31.195992 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:31.197995 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:31.197998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:31.198000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:31.198022 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:31.199356 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:31.199856 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:31.199858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:31.199859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:31.199962 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:31.200296 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:31.200299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:31.200301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:31.200470 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:31.200542 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:31.200923 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 14:50:31.201006 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:50:31.201797 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:31.191367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:38073 (local address 127.12.68.195:57149)
0926 14:50:31.191747 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:31.191792 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:31.192925 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:31.193350 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:31.193356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:31.193663 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:31.194108 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:31.194120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:31.194798 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:31.194800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:31.195612 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:31.195620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:31.196023 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:31.196026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:31.196151 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:31.196569 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:31.196583 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:31.197843 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:31.199445 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:31.199449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:31.199459 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:31.199774 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:31.200042 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:31.200044 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:31.200046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:31.200209 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:31.200542 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:31.200546 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:31.200752 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 14:50:31.200892 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 14:50:31.201102 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
I20260926 14:50:31.202705 18607 heartbeater.cc:347] Connected to a master server at 127.12.68.254:38073
I20260926 14:50:31.202879 18607 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:31.203259 18607 heartbeater.cc:511] Master 127.12.68.254:38073 requested a full tablet report, sending...
I20260926 14:50:31.204105 18148 ts_manager.cc:194] Registered new tserver with Master: 0a438ca4cc444902b96e44aee0f68662 (127.12.68.195:38657)
I20260926 14:50:31.204595 18148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:57149
I20260926 14:50:31.211453 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/test-admin.keytab.
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:31.223680 12563 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 14:50:31.226193 18614 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:31.224917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.12.68.193:36769)
0926 14:50:31.225188 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:31.225199 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:31.225229 (+ 30us) server_negotiation.cc:407] Connection header received
0926 14:50:31.225272 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:31.225277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:31.225436 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:31.225584 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:31.226035 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34610: BlockingRecv error: recv got EOF from 127.0.0.1:34610 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
W20260926 14:50:31.226253 12563 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/krb5cc' of type FILE
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:31.227142 12563 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:31.227998 12563 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.193@KRBTEST.COM
I20260926 14:50:31.236054 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:31.228733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.12.68.193:36769)
0926 14:50:31.228850 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:31.228854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:31.228918 (+ 64us) server_negotiation.cc:407] Connection header received
0926 14:50:31.229001 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:31.229004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:31.229038 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:31.229101 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:31.229515 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:31.230232 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:31.230997 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:31.231342 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:31.231415 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:31.233385 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:31.233393 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:31.233404 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:31.233426 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:31.234676 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:31.235006 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:31.235010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:31.235011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:31.235076 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:31.235247 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:31.235249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:31.235250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:31.235455 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:31.235595 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:31.235758 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 14:50:31.235903 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 14:50:31.239265 18614 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:31.238405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.12.68.193:36769)
0926 14:50:31.238517 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:31.238521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:31.238589 (+ 68us) server_negotiation.cc:407] Connection header received
0926 14:50:31.238690 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:31.238693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:31.238727 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:31.238794 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:31.239182 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34618: BlockingRecv error: recv got EOF from 127.0.0.1:34618 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 14:50:31.239393 12563 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/krb5cc' of type FILE
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:31.240258 12563 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:31.241055 12563 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.193@KRBTEST.COM
I20260926 14:50:31.247761 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:31.241702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.12.68.193:36769)
0926 14:50:31.241809 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:31.241811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:31.241859 (+ 48us) server_negotiation.cc:407] Connection header received
0926 14:50:31.241937 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:31.241939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:31.241969 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:31.242023 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:31.242394 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:31.242890 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:31.243548 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:31.243747 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:31.243807 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:31.245616 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:31.245619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:31.245621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:31.245640 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:31.246646 (+ 1006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:31.246967 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:31.246969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:31.246970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:31.247028 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:31.247196 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:31.247198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:31.247199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:31.247329 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:31.247408 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:31.247555 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 14:50:31.247631 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 14:50:31.250034 18614 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:31.249226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.12.68.193:36769)
0926 14:50:31.249338 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:31.249341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:31.249411 (+ 70us) server_negotiation.cc:407] Connection header received
0926 14:50:31.249508 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:31.249511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:31.249545 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:31.249605 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:31.249953 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34636: BlockingRecv error: recv got EOF from 127.0.0.1:34636 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 14:50:31.250108 12563 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434204424021-12563-0/krb5kdc/krb5cc' of type FILE
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:31.250943 12563 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:31.251689 12563 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.193@KRBTEST.COM
I20260926 14:50:31.258975 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:31.252399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.12.68.193:36769)
0926 14:50:31.252493 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:31.252497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:31.252565 (+ 68us) server_negotiation.cc:407] Connection header received
0926 14:50:31.252653 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:31.252656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:31.252709 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:31.252767 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:31.253219 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:31.253675 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:31.254394 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:31.254611 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:31.254691 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:31.256604 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:31.256607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:31.256610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:31.256633 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:31.257628 (+ 995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:31.257984 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:31.257988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:31.257989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:31.258061 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:31.258284 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:31.258287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:31.258288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:31.258434 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:31.258534 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:31.258742 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 14:50:31.258844 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:50:31.259824 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 18210
I20260926 14:50:31.264856 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 18344
I20260926 14:50:31.269711 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 18478
I20260926 14:50:31.273754 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 18116
2026-09-26T14:50:31Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1966 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18627](info): set up 2 sockets
Sep 26 14:50:31 dist-test-slave-24mn krb5kdc[18627](info): commencing operation
krb5kdc: starting...
W20260926 14:50:32.232003 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:50:32 dist-test-slave-24mn krb5kdc[18627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.254.keytab.
Entry for principal oryx/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.254.keytab.
I20260926 14:50:32.340451 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:43839
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.254.keytab
--principal=oryx/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:43839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:32.442783 18640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:32.443015 18640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:32.443068 18640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:32.446993 18640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:32.447042 18640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:32.447068 18640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:32.447090 18640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:32.447124 18640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:32.452169 18640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:43839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.254.keytab
--principal=oryx/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:43839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:32.453065 18640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:32.453977 18640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:32.461601 18646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:32.461613 18648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:32.461596 18645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:32.461946 18640 server_base.cc:1034] running on GCE node
I20260926 14:50:32.462440 18640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:32.463425 18640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:32.464587 18640 hybrid_clock.cc:648] HybridClock initialized: now 1790434232464565 us; error 34 us; skew 500 ppm
Sep 26 14:50:32 dist-test-slave-24mn krb5kdc[18627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434232, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:32.467057 18640 init.cc:377] Logged in from keytab as oryx/127.12.68.254@KRBTEST.COM (short username oryx)
I20260926 14:50:32.467818 18640 webserver.cc:460] Webserver started at http://127.12.68.254:39179/ using document root <none> and password file <none>
I20260926 14:50:32.468317 18640 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:32.468390 18640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:32.468591 18640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:32.470469 18640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a"
format_stamp: "Formatted at 2026-09-26 14:50:32 on dist-test-slave-24mn"
server_key: "5ee8f03dc9e3342f00ec87a1b5abf142"
server_key_iv: "24e0ec470e5ebe6651f073a399ee92a4"
server_key_version: "encryptionkey@0"
I20260926 14:50:32.470968 18640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a"
format_stamp: "Formatted at 2026-09-26 14:50:32 on dist-test-slave-24mn"
server_key: "5ee8f03dc9e3342f00ec87a1b5abf142"
server_key_iv: "24e0ec470e5ebe6651f073a399ee92a4"
server_key_version: "encryptionkey@0"
I20260926 14:50:32.474288 18640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:50:32.476459 18655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:32.477548 18640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:32.477694 18640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a"
format_stamp: "Formatted at 2026-09-26 14:50:32 on dist-test-slave-24mn"
server_key: "5ee8f03dc9e3342f00ec87a1b5abf142"
server_key_iv: "24e0ec470e5ebe6651f073a399ee92a4"
server_key_version: "encryptionkey@0"
I20260926 14:50:32.477780 18640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:32.499158 18640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:32.501816 18640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:32.501978 18640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:32.508117 18640 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:43839
I20260926 14:50:32.508353 18707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:43839 every 8 connection(s)
I20260926 14:50:32.509027 18640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:32.511586 18708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:32.515101 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 18640
I20260926 14:50:32.515208 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:32.515458 12563 external_mini_cluster.cc:1428] Setting key 74c2da17e3c91e052ac6ad8b9f81db68
I20260926 14:50:32.516887 18708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a: Bootstrap starting.
I20260926 14:50:32.519377 18708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:32.520215 18708 log.cc:824] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:32.521901 18708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a: No bootstrap required, opened a new log
Sep 26 14:50:32 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
I20260926 14:50:32.525653 18708 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } }
I20260926 14:50:32.525864 18708 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:32.525926 18708 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf70cbde5ec4195a72786c4dd8b3e5a, State: Initialized, Role: FOLLOWER
I20260926 14:50:32.526337 18708 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } }
I20260926 14:50:32.526463 18708 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:32.526542 18708 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:32.526643 18708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:32.527611 18708 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } }
I20260926 14:50:32.527987 18708 leader_election.cc:302] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccf70cbde5ec4195a72786c4dd8b3e5a; no voters:
I20260926 14:50:32.528292 18708 leader_election.cc:288] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:32.528342 18713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:32.528548 18713 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [term 1 LEADER]: Becoming Leader. State: Replica: ccf70cbde5ec4195a72786c4dd8b3e5a, State: Running, Role: LEADER
I20260926 14:50:32.528868 18713 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } }
I20260926 14:50:32.529013 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.516754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.12.68.254:43839)
0926 14:50:32.517175 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:32.517193 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:32.517234 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:50:32.518020 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:32.518042 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:32.518428 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:32.518766 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:32.519360 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.520755 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.521562 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.522005 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.522128 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.524622 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:32.524626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:32.524638 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:32.524663 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:32.526166 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:32.526538 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:32.526543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:32.526549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:32.526685 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:32.526936 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:32.526940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:32.526941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:32.527412 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:32.527596 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:32.527796 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 14:50:32.528103 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":102,"threads_started":1}
I20260926 14:50:32.529933 18713 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccf70cbde5ec4195a72786c4dd8b3e5a. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } }
I20260926 14:50:32.530051 18713 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:32.530393 18714 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } }
I20260926 14:50:32.530465 18714 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:32.531203 18708 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:32.531628 18717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:32.534463 18717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:32.539332 18717 catalog_manager.cc:1347] Generated new cluster ID: 71d27facd073479e96c0cc5b21759065
I20260926 14:50:32.539381 18717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:32.551738 18717 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:32.552681 18717 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:32.570930 18717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a: Generated new TSK 0
I20260926 14:50:32.571521 18717 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.193.keytab.
Entry for principal oryx/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.193.keytab.
I20260926 14:50:32.602880 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43839
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.193.keytab
--principal=oryx/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:32.709869 18734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:32.710083 18734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:32.710134 18734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:32.714031 18734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:32.714078 18734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:32.714205 18734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:32.719316 18734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.193.keytab
--principal=oryx/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:32.720201 18734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:32.721074 18734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:32.730087 18740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:32.730094 18742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:32.730094 18739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:32.730356 18734 server_base.cc:1034] running on GCE node
I20260926 14:50:32.730703 18734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:32.731266 18734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:32.732426 18734 hybrid_clock.cc:648] HybridClock initialized: now 1790434232732406 us; error 31 us; skew 500 ppm
Sep 26 14:50:32 dist-test-slave-24mn krb5kdc[18627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434232, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:32.734902 18734 init.cc:377] Logged in from keytab as oryx/127.12.68.193@KRBTEST.COM (short username oryx)
I20260926 14:50:32.735628 18734 webserver.cc:460] Webserver started at http://127.12.68.193:43855/ using document root <none> and password file <none>
I20260926 14:50:32.736140 18734 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:32.736196 18734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:32.736352 18734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:32.738173 18734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "9c41ef8146da478c86767ede40d05247"
format_stamp: "Formatted at 2026-09-26 14:50:32 on dist-test-slave-24mn"
server_key: "97645633095fd9768184af480221e48f"
server_key_iv: "6b752c71e48e29c55984fb8ebe09686e"
server_key_version: "encryptionkey@0"
I20260926 14:50:32.738641 18734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "9c41ef8146da478c86767ede40d05247"
format_stamp: "Formatted at 2026-09-26 14:50:32 on dist-test-slave-24mn"
server_key: "97645633095fd9768184af480221e48f"
server_key_iv: "6b752c71e48e29c55984fb8ebe09686e"
server_key_version: "encryptionkey@0"
I20260926 14:50:32.741923 18734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:50:32.744042 18749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:32.745095 18734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:32.745188 18734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "9c41ef8146da478c86767ede40d05247"
format_stamp: "Formatted at 2026-09-26 14:50:32 on dist-test-slave-24mn"
server_key: "97645633095fd9768184af480221e48f"
server_key_iv: "6b752c71e48e29c55984fb8ebe09686e"
server_key_version: "encryptionkey@0"
I20260926 14:50:32.745260 18734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:32.761053 18734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:32.763401 18734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:32.763528 18734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:32.764002 18734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:32.764693 18734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:32.764737 18734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:32.764779 18734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:32.764801 18734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:32.772315 18734 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:38767
I20260926 14:50:32.772365 18862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:38767 every 8 connection(s)
I20260926 14:50:32.773221 18734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:32.778491 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 18734
I20260926 14:50:32.778628 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:32.778944 12563 external_mini_cluster.cc:1428] Setting key bd4e7c192375f35cabae8562280bcea5
Sep 26 14:50:32 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434232, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
I20260926 14:50:32.786221 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.774823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:40153 (local address 127.12.68.254:43839)
0926 14:50:32.774934 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:32.774938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:32.775693 (+ 755us) server_negotiation.cc:407] Connection header received
0926 14:50:32.776966 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:32.776969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:32.777015 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:32.777075 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:32.778081 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.778754 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.779983 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.780270 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.780373 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.783104 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:32.783107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:32.783109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:32.783132 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:32.784411 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:32.784862 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:32.784865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:32.784867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:32.784941 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:32.785242 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:32.785245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:32.785247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:32.785413 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:32.785493 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:32.785952 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 14:50:32.786067 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:50:32.786953 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.775123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.193:40153)
0926 14:50:32.775550 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.775589 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:32.776773 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:32.777207 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:32.777213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:32.777528 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:32.777976 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:32.777989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.778894 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.778896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:32.779863 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:32.779871 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.780444 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.780448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:32.780623 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.781147 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:32.781193 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:32.782925 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:32.784514 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:32.784520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:32.784533 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:32.784771 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:32.785018 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:32.785021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:32.785023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:32.785159 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:32.785506 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:32.785512 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:32.785741 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 14:50:32.785898 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 14:50:32.786202 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for oryx/127.12.68.194@KRBTEST.COM; defaulting to no policy
I20260926 14:50:32.788084 18863 heartbeater.cc:347] Connected to a master server at 127.12.68.254:43839
I20260926 14:50:32.788282 18863 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:32.788712 18863 heartbeater.cc:511] Master 127.12.68.254:43839 requested a full tablet report, sending...
I20260926 14:50:32.790149 18672 ts_manager.cc:194] Registered new tserver with Master: 9c41ef8146da478c86767ede40d05247 (127.12.68.193:38767)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.68.194@KRBTEST.COM" created.
I20260926 14:50:32.791254 18672 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:40153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.194.keytab.
Entry for principal oryx/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.194.keytab.
I20260926 14:50:32.806623 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43839
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.194.keytab
--principal=oryx/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:32.910192 18868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:32.910427 18868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:32.910467 18868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:32.914438 18868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:32.914489 18868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:32.914546 18868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:32.919505 18868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.194.keytab
--principal=oryx/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:32.920454 18868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:32.921394 18868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:32.930164 18873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:32.930230 18876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:32.930166 18874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:32.930370 18868 server_base.cc:1034] running on GCE node
I20260926 14:50:32.930845 18868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:32.931414 18868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:32.932634 18868 hybrid_clock.cc:648] HybridClock initialized: now 1790434232932613 us; error 32 us; skew 500 ppm
Sep 26 14:50:32 dist-test-slave-24mn krb5kdc[18627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434232, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:32.935089 18868 init.cc:377] Logged in from keytab as oryx/127.12.68.194@KRBTEST.COM (short username oryx)
I20260926 14:50:32.935896 18868 webserver.cc:460] Webserver started at http://127.12.68.194:44259/ using document root <none> and password file <none>
I20260926 14:50:32.936389 18868 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:32.936444 18868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:32.936651 18868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:32.938484 18868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "f7b60ac486fb400e95bb50df4ed3eb98"
format_stamp: "Formatted at 2026-09-26 14:50:32 on dist-test-slave-24mn"
server_key: "2b529ce575fcff41f21d9bc69121d997"
server_key_iv: "cc4d2be2d2352c688aef8e89979a146f"
server_key_version: "encryptionkey@0"
I20260926 14:50:32.938987 18868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "f7b60ac486fb400e95bb50df4ed3eb98"
format_stamp: "Formatted at 2026-09-26 14:50:32 on dist-test-slave-24mn"
server_key: "2b529ce575fcff41f21d9bc69121d997"
server_key_iv: "cc4d2be2d2352c688aef8e89979a146f"
server_key_version: "encryptionkey@0"
I20260926 14:50:32.942343 18868 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:50:32.944573 18883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:32.945680 18868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:32.945777 18868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "f7b60ac486fb400e95bb50df4ed3eb98"
format_stamp: "Formatted at 2026-09-26 14:50:32 on dist-test-slave-24mn"
server_key: "2b529ce575fcff41f21d9bc69121d997"
server_key_iv: "cc4d2be2d2352c688aef8e89979a146f"
server_key_version: "encryptionkey@0"
I20260926 14:50:32.945859 18868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:32.966470 18868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:32.969274 18868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:32.969434 18868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:32.969972 18868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:32.970736 18868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:32.970781 18868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:32.970829 18868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:32.970858 18868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:32.978737 18868 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:45333
I20260926 14:50:32.978765 18996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:45333 every 8 connection(s)
I20260926 14:50:32.979530 18868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:32.981930 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 18868
I20260926 14:50:32.982033 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:32.982339 12563 external_mini_cluster.cc:1428] Setting key 0178b6cf5fd6d56bd837b1ecbb0bf3bd
Sep 26 14:50:32 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434232, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.12.68.195@KRBTEST.COM; defaulting to no policy
I20260926 14:50:32.993140 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.981141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:37583 (local address 127.12.68.254:43839)
0926 14:50:32.981255 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:32.981260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:32.982153 (+ 893us) server_negotiation.cc:407] Connection header received
0926 14:50:32.983584 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:32.983587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:32.983636 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:32.983721 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:32.985089 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.985607 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.987114 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.987329 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.987402 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.989969 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:32.989972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:32.989974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:32.989997 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:32.991231 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:32.991763 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:32.991766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:32.991768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:32.991846 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:32.992176 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:32.992179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:32.992180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:32.992391 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:32.992482 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:32.992888 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 14:50:32.992987 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.68.195@KRBTEST.COM" created.
I20260926 14:50:32.993896 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.981582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.194:37583)
0926 14:50:32.982015 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.982063 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:32.983387 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:32.983914 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:32.983922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:32.984354 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:32.984966 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:32.984983 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.985727 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.985731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:32.987002 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:32.987012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.987465 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.987469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:32.987657 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.988177 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:32.988197 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:32.989811 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:32.991341 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:32.991347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:32.991362 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:32.991671 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:32.991968 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:32.991970 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:32.991971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:32.992103 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:32.992479 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:32.992484 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:32.992695 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 14:50:32.992841 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 14:50:32.993066 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":90,"threads_started":1}
I20260926 14:50:32.994985 18997 heartbeater.cc:347] Connected to a master server at 127.12.68.254:43839
I20260926 14:50:32.995199 18997 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:32.995640 18997 heartbeater.cc:511] Master 127.12.68.254:43839 requested a full tablet report, sending...
I20260926 14:50:32.996644 18672 ts_manager.cc:194] Registered new tserver with Master: f7b60ac486fb400e95bb50df4ed3eb98 (127.12.68.194:45333)
I20260926 14:50:32.997241 18672 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:37583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.195.keytab.
Entry for principal oryx/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.195.keytab.
I20260926 14:50:33.008191 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43839
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.195.keytab
--principal=oryx/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:33.112548 19002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:33.112782 19002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:33.112838 19002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:33.116732 19002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:33.116787 19002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:33.116842 19002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:33.121733 19002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.195.keytab
--principal=oryx/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:33.122591 19002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:33.123458 19002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:33.132537 19008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:33.132596 19007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:33.132659 19002 server_base.cc:1034] running on GCE node
W20260926 14:50:33.132611 19010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:33.133280 19002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:33.133852 19002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:33.134989 19002 hybrid_clock.cc:648] HybridClock initialized: now 1790434233134966 us; error 40 us; skew 500 ppm
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:33.137504 19002 init.cc:377] Logged in from keytab as oryx/127.12.68.195@KRBTEST.COM (short username oryx)
I20260926 14:50:33.138247 19002 webserver.cc:460] Webserver started at http://127.12.68.195:36097/ using document root <none> and password file <none>
I20260926 14:50:33.138737 19002 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:33.138798 19002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:33.139027 19002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:33.140951 19002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "687fc8c182854d2da4f7d2a7418ce476"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-24mn"
server_key: "4985b000d1ac09645d249b6fad13d596"
server_key_iv: "4cd716d4f3143d5d21ce4c3a808be058"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.141443 19002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "687fc8c182854d2da4f7d2a7418ce476"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-24mn"
server_key: "4985b000d1ac09645d249b6fad13d596"
server_key_iv: "4cd716d4f3143d5d21ce4c3a808be058"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.144819 19002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:33.146919 19017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:33.148042 19002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:50:33.148137 19002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "687fc8c182854d2da4f7d2a7418ce476"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-24mn"
server_key: "4985b000d1ac09645d249b6fad13d596"
server_key_iv: "4cd716d4f3143d5d21ce4c3a808be058"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.148254 19002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:33.162988 19002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:33.166006 19002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:33.166178 19002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:33.166788 19002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:33.167611 19002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:33.167654 19002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:33.167690 19002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:33.167711 19002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:33.175418 19002 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:44785
I20260926 14:50:33.175446 19130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:44785 every 8 connection(s)
I20260926 14:50:33.176218 19002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:33.183280 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 19002
I20260926 14:50:33.183391 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:33.183676 12563 external_mini_cluster.cc:1428] Setting key 63af9a2afb86234e770eb1458739ffbc
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
I20260926 14:50:33.188566 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.177814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:41141 (local address 127.12.68.254:43839)
0926 14:50:33.177951 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.177955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.178686 (+ 731us) server_negotiation.cc:407] Connection header received
0926 14:50:33.179985 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.179988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.180042 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.180114 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.181229 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.181769 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.182841 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.183050 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.183148 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.185737 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.185741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.185744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.185770 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.186974 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.187394 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.187396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.187397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.187461 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.187695 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.187697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.187698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.187846 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.187960 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.188353 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.188443 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:50:33.189306 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.178163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.195:41141)
0926 14:50:33.178552 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.178591 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.179779 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.180284 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.180293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.180637 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.181093 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.181106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.181880 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.181883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.182713 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.182722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.183181 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.183184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.183322 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.183858 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.183873 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.185583 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.187068 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.187073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.187084 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.187319 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.187520 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.187522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.187523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.187634 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.187952 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.187958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.188168 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.188319 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.188555 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":91,"threads_started":1}
I20260926 14:50:33.190243 19131 heartbeater.cc:347] Connected to a master server at 127.12.68.254:43839
I20260926 14:50:33.190449 19131 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:33.190871 19131 heartbeater.cc:511] Master 127.12.68.254:43839 requested a full tablet report, sending...
I20260926 14:50:33.191732 18672 ts_manager.cc:194] Registered new tserver with Master: 687fc8c182854d2da4f7d2a7418ce476 (127.12.68.195:44785)
I20260926 14:50:33.192270 18672 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:41141
I20260926 14:50:33.197137 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.68.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.68.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.68.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.253.keytab.
Entry for principal oryx/127.12.68.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.253.keytab.
I20260926 14:50:33.218729 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.253:44171
--webserver_interface=127.12.68.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.253.keytab
--principal=oryx/127.12.68.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:43839,127.12.68.253:44171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:33.322751 19136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:33.323011 19136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:33.323074 19136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:33.327090 19136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:33.327145 19136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:33.327171 19136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:33.327189 19136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:33.327204 19136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:33.332592 19136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:43839,127.12.68.253:44171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.253.keytab
--principal=oryx/127.12.68.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.253:44171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.68.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
W20260926 14:50:33.332754 19136 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.68.254:43839,127.12.68.253:44171'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 14:50:33.333485 19136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:33.334370 19136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:33.342095 19141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:33.342073 19144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:33.342142 19136 server_base.cc:1034] running on GCE node
W20260926 14:50:33.342067 19142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:33.342730 19136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:33.343770 19136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:33.344918 19136 hybrid_clock.cc:648] HybridClock initialized: now 1790434233344892 us; error 37 us; skew 500 ppm
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:33.347313 19136 init.cc:377] Logged in from keytab as oryx/127.12.68.253@KRBTEST.COM (short username oryx)
I20260926 14:50:33.348163 19136 webserver.cc:460] Webserver started at http://127.12.68.253:32813/ using document root <none> and password file <none>
I20260926 14:50:33.348773 19136 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:33.348856 19136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:33.349044 19136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:33.351001 19136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/data/instance:
uuid: "e7d0f6987f9e42e586b68f89d0bc95b4"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-24mn"
server_key: "79b266ec8e1df5efdebaa387b9df4620"
server_key_iv: "5ea4f121d674c78bb91f6b7783312f95"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.351528 19136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/wal/instance:
uuid: "e7d0f6987f9e42e586b68f89d0bc95b4"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-24mn"
server_key: "79b266ec8e1df5efdebaa387b9df4620"
server_key_iv: "5ea4f121d674c78bb91f6b7783312f95"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.354938 19136 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:50:33.357112 19151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:33.358215 19136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:33.358320 19136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/wal
uuid: "e7d0f6987f9e42e586b68f89d0bc95b4"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-24mn"
server_key: "79b266ec8e1df5efdebaa387b9df4620"
server_key_iv: "5ea4f121d674c78bb91f6b7783312f95"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.358405 19136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:33.372114 19136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:33.374531 19136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:33.374681 19136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:33.380765 19136 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.253:44171
I20260926 14:50:33.380987 19203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.253:44171 every 8 connection(s)
I20260926 14:50:33.381649 19136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/data/info.pb
I20260926 14:50:33.383801 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 19136
I20260926 14:50:33.384392 19204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:33.388960 19204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.68.253@KRBTEST.COM
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.253@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
I20260926 14:50:33.397552 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.385211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49080 (local address 127.12.68.253:44171)
0926 14:50:33.385583 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.385601 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.385640 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:50:33.386386 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.386408 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.386800 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.387168 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.387778 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.389231 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.390067 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.390583 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.390731 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.393168 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.393174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.393184 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.393213 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.395000 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.395456 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.395461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.395466 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.395570 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.395821 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.395824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.395825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.396258 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.396434 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.396661 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.396890 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20260926 14:50:33.399078 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.390614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55550 (local address 127.12.68.254:43839)
0926 14:50:33.390744 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.390748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.391041 (+ 293us) server_negotiation.cc:407] Connection header received
0926 14:50:33.391232 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.391236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.391314 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.391400 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.392029 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.392518 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.393533 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.393834 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.393918 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.396144 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.396147 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.396149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.396171 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.397484 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.397933 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.397936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.397937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.398010 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.398242 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.398244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.398245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.398432 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.398515 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.398835 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.398937 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 14:50:33.399088 19208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.390530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.0.0.1:55550)
0926 14:50:33.390942 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.390976 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.391121 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.391443 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.391447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.391757 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.391868 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.391873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.392636 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.392638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.393408 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.393417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.393956 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.393960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.394069 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.394409 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.394423 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.396028 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.397582 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.397588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.397593 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.397849 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.398084 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.398087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.398088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.398174 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.398533 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.398539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.398638 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.398750 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.398913 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":100,"threads_started":1}
I20260926 14:50:33.400131 19204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.253@KRBTEST.COM for oryx/127.12.68.253@KRBTEST.COM
I20260926 14:50:33.406668 19208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.400489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44171 (local address 127.0.0.1:49084)
0926 14:50:33.400567 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.400579 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.400645 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.400826 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.400828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.400959 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.401030 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.401034 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.401989 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.401992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.402479 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.402485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.402888 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.402891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.402966 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.403282 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.403292 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.404329 (+ 1037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.405510 (+ 1181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.405513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.405515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.405672 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.405895 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.405898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.405900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.405963 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.406257 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.406260 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.406329 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.406407 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.406520 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 14:50:33.406685 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.400539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.12.68.253:44171)
0926 14:50:33.400627 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.400630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.400642 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:33.400708 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.400710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.400742 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.400808 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.401119 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.401884 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.402572 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.402789 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.402867 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.404434 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.404437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.404439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.404456 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.405422 (+ 966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.405745 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.405748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.405763 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.405822 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.406032 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.406035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.406036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.406170 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.406245 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.406456 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.406570 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":36}
I20260926 14:50:33.409159 19204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: Bootstrap starting.
I20260926 14:50:33.411128 19204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:33.411757 19204 log.cc:824] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:33.413389 19204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: No bootstrap required, opened a new log
I20260926 14:50:33.416360 19204 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } }
I20260926 14:50:33.416543 19204 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:33.416605 19204 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d0f6987f9e42e586b68f89d0bc95b4, State: Initialized, Role: FOLLOWER
I20260926 14:50:33.416965 19204 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } }
I20260926 14:50:33.417322 19210 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } } }
I20260926 14:50:33.417419 19210 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:50:33.417944 19204 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 14:50:33.421965 19221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:50:33.422039 19221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:50:33.423224 19136 master_runner.cc:428] Detected that this master e7d0f6987f9e42e586b68f89d0bc95b4 is joining an existing cluster
I20260926 14:50:33.423269 19136 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.68.253:44171
I20260926 14:50:33.432024 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.424942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.0.0.1:55554)
0926 14:50:33.425282 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.425358 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.425447 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.425663 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.425666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.425853 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.425985 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.425992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.426854 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.426857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.427504 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.427511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.427941 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.427945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.428048 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.428491 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.428504 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.428855 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.430514 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.430517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.430519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.430714 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.431013 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.431016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.431018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.431094 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.431528 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.431531 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.431603 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.431711 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.431825 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":21,"thread_start_us":90,"threads_started":1}
I20260926 14:50:33.432224 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.424996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.12.68.254:43839)
0926 14:50:33.425130 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.425134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.425425 (+ 291us) server_negotiation.cc:407] Connection header received
0926 14:50:33.425528 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.425530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.425566 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.425633 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.426069 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.426743 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.427611 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.427828 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.427937 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.428996 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.429000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.429003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.429027 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.430402 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.430821 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.430825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.430827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.430907 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.431205 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.431208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.431210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.431407 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.431468 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.431977 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.432081 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:50:33.433261 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.424966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44171 (local address 127.0.0.1:49086)
0926 14:50:33.425628 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.425639 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.425734 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.425967 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.425970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.426172 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.426279 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.426285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.427153 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.427156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.428090 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.428101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.428620 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.428624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.428713 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.429106 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.429120 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.429526 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.431398 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.431401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.431403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.431659 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.431935 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.431938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.431940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.432068 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.432742 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.432745 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.432830 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.432950 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.433072 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260926 14:50:33.433277 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.425268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.12.68.253:44171)
0926 14:50:33.425334 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.425338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.425690 (+ 352us) server_negotiation.cc:407] Connection header received
0926 14:50:33.425822 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.425824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.425862 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.425937 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.426383 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.427036 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.428200 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.428498 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.428617 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.429856 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.429859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.429862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.429885 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.431308 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.431734 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.431737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.431738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.431809 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.432398 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.432401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.432403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.432651 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.432770 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.433006 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.433135 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":23}
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434232, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.254@KRBTEST.COM for oryx/127.12.68.253@KRBTEST.COM
I20260926 14:50:33.445101 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.436932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44171 (local address 127.0.0.1:49090)
0926 14:50:33.437334 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.437379 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.437501 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.437761 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.437765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.438052 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.438161 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.438167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.439076 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.439078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.439684 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.439692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.440201 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.440204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.440283 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.440715 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.440726 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.441975 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.443559 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.443567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.443570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.443770 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.444069 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.444072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.444075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.444175 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.444600 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.444605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.444724 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.444844 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.444979 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":71,"threads_started":1}
I20260926 14:50:33.445132 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.436947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.12.68.253:44171)
0926 14:50:33.437032 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.437036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.437451 (+ 415us) server_negotiation.cc:407] Connection header received
0926 14:50:33.437617 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.437620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.437664 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.437741 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.438249 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.438974 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.439773 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.440111 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.440197 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.442098 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.442102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.442105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.442128 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.443452 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.443849 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.443852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.443854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.443949 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.444264 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.444267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.444269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.444481 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.444570 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.444901 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.444995 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:50:33.446370 18672 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.68.253:44171: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4"
member_type: NON_VOTER
last_known_addr {
host: "127.12.68.253"
port: 44171
}
attrs {
promote: true
}
}
dest_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a"
cas_config_opid_index: -1
I20260926 14:50:33.446908 18672 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: NON_VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: true } }
I20260926 14:50:33.448287 19231 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e7d0f6987f9e42e586b68f89d0bc95b4 (127.12.68.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: NON_VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: true } } }
I20260926 14:50:33.449043 19178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:33.449899 19231 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.68.253:44171
I20260926 14:50:33.450129 19231 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: NON_VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: true } } }
I20260926 14:50:33.450251 19231 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:33.450361 19178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 FOLLOWER]: Refusing update from remote peer ccf70cbde5ec4195a72786c4dd8b3e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 14:50:33.451152 19231 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: NON_VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:33.451288 19136 master.cc:537] Master@127.12.68.253:44171 shutting down...
W20260926 14:50:33.453243 18656 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 14:50:33.454026 18656 consensus_peers.cc:487] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a -> Peer e7d0f6987f9e42e586b68f89d0bc95b4 (127.12.68.253:44171): Couldn't send request to peer e7d0f6987f9e42e586b68f89d0bc95b4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 14:50:33.455339 19136 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:33.455559 19136 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:33.455657 19136 tablet_replica.cc:330] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: stopping tablet replica
I20260926 14:50:33.469429 19136 master.cc:552] Master@127.12.68.253:44171 shutdown complete.
I20260926 14:50:33.469471 19136 master_runner.cc:305] Clearing existing system tablet
I20260926 14:50:33.469671 19136 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:33.469717 19136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
WARNING: no policy specified for oryx/127.12.68.252@KRBTEST.COM; defaulting to no policy
I20260926 14:50:33.470692 19136 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:33.471366 19236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:33.471621 19136 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:33.471740 19136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:33.471779 19136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/wal
uuid: "e7d0f6987f9e42e586b68f89d0bc95b4"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-24mn"
server_key: "79b266ec8e1df5efdebaa387b9df4620"
server_key_iv: "5ea4f121d674c78bb91f6b7783312f95"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.68.252@KRBTEST.COM" created.
I20260926 14:50:33.474292 19136 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:33.476549 19136 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 14:50:33.476642 19136 log.cc:1196] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 14:50:33.476907 19136 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: Deleting consensus metadata
I20260926 14:50:33.477056 19136 master_runner.cc:315] Copying system tablet from 127.12.68.254:43839
I20260926 14:50:33.478094 19136 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: tablet copy: Beginning tablet copy session from remote peer at address 127.12.68.254:43839
I20260926 14:50:33.485816 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.478690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55562 (local address 127.12.68.254:43839)
0926 14:50:33.478804 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.478809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.478974 (+ 165us) server_negotiation.cc:407] Connection header received
0926 14:50:33.479096 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.479099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.479146 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.479235 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.479711 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.480549 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.481378 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.481719 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.481808 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.483006 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.483009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.483011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.483049 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.484387 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.484811 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.484813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.484814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.484882 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.485112 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.485114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.485116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.485270 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.485339 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.485621 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.485707 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:50:33.485852 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.478583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.0.0.1:55562)
0926 14:50:33.478887 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.478901 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.479009 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.479266 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.479269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.479457 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.479589 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.479608 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.480649 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.480652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.481251 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.481262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.481942 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.481946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.482039 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.482491 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.482504 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.482883 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.484487 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.484491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.484493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.484729 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.484953 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.484956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.484957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.485038 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.485342 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.485345 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.485433 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.485529 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.485659 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":79,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.68.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.252.keytab.
Entry for principal oryx/127.12.68.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.252.keytab.
I20260926 14:50:33.486615 18692 tablet_copy_service.cc:140] P ccf70cbde5ec4195a72786c4dd8b3e5a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e7d0f6987f9e42e586b68f89d0bc95b4 ({username='oryx', principal='oryx/127.12.68.253@KRBTEST.COM'} at 127.0.0.1:55562)
I20260926 14:50:33.486786 18692 tablet_copy_service.cc:161] P ccf70cbde5ec4195a72786c4dd8b3e5a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e7d0f6987f9e42e586b68f89d0bc95b4 at {username='oryx', principal='oryx/127.12.68.253@KRBTEST.COM'} at 127.0.0.1:55562: session id = e7d0f6987f9e42e586b68f89d0bc95b4-00000000000000000000000000000000
I20260926 14:50:33.486932 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.252:46837
--webserver_interface=127.12.68.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.252.keytab
--principal=oryx/127.12.68.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:43839,127.12.68.253:44171,127.12.68.252:46837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:50:33.488418 18692 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 14:50:33.489789 19136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 14:50:33.491858 19136 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: tablet copy: Starting download of 0 data blocks...
I20260926 14:50:33.492051 19136 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: tablet copy: Starting download of 1 WAL segments...
I20260926 14:50:33.493631 19136 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 14:50:33.495234 18692 tablet_copy_service.cc:342] P ccf70cbde5ec4195a72786c4dd8b3e5a: Request end of tablet copy session e7d0f6987f9e42e586b68f89d0bc95b4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.68.253@KRBTEST.COM'} at 127.0.0.1:55562
I20260926 14:50:33.495347 18692 tablet_copy_service.cc:433] P ccf70cbde5ec4195a72786c4dd8b3e5a: ending tablet copy session e7d0f6987f9e42e586b68f89d0bc95b4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e7d0f6987f9e42e586b68f89d0bc95b4
I20260926 14:50:33.502445 19136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:33.503957 19245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:33.504139 19246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:33.504319 19248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:33.510988 19136 server_base.cc:1034] running on GCE node
I20260926 14:50:33.511186 19136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:33.511435 19136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:33.512557 19136 hybrid_clock.cc:648] HybridClock initialized: now 1790434233512543 us; error 26 us; skew 500 ppm
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:33.514288 19136 init.cc:377] Logged in from keytab as oryx/127.12.68.253@KRBTEST.COM (short username oryx)
I20260926 14:50:33.514734 19136 webserver.cc:460] Webserver started at http://127.12.68.253:46235/ using document root <none> and password file <none>
I20260926 14:50:33.514899 19136 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:33.514941 19136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:33.516124 19136 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:33.516994 19255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:33.517185 19136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:33.517246 19136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/wal
uuid: "e7d0f6987f9e42e586b68f89d0bc95b4"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-24mn"
server_key: "79b266ec8e1df5efdebaa387b9df4620"
server_key_iv: "5ea4f121d674c78bb91f6b7783312f95"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.517316 19136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:33.548719 19136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:33.550853 19136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:33.555744 19136 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.253:44171
I20260926 14:50:33.555778 19307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.253:44171 every 8 connection(s)
I20260926 14:50:33.556332 19136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-1/data/info.pb
I20260926 14:50:33.558059 19308 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:50:33.559062 19308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: Bootstrap starting.
I20260926 14:50:33.568799 19308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:50:33.569062 19308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: Bootstrap complete.
I20260926 14:50:33.569588 19308 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: NON_VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: true } }
I20260926 14:50:33.569720 19308 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e7d0f6987f9e42e586b68f89d0bc95b4, State: Initialized, Role: LEARNER
I20260926 14:50:33.569821 19308 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: NON_VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: true } }
I20260926 14:50:33.570114 19311 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: NON_VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: true } } }
I20260926 14:50:33.570286 19311 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:50:33.570297 19308 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:33.573006 19322 catalog_manager.cc:1259] Loaded cluster ID: 71d27facd073479e96c0cc5b21759065
I20260926 14:50:33.573103 19322 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: loading cluster ID for follower catalog manager: success
I20260926 14:50:33.575158 19322 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: acquiring CA information for follower catalog manager: success
I20260926 14:50:33.576447 19322 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 14:50:33.608189 19244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:33.608419 19244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:33.608472 19244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:33.612325 19244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:33.612375 19244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:33.612399 19244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:33.612428 19244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:33.612454 19244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:33.617667 19244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:43839,127.12.68.253:44171,127.12.68.252:46837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.252.keytab
--principal=oryx/127.12.68.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.252:46837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.68.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:33.618631 19244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:33.619552 19244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:33.626921 19327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:33.626997 19328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:33.627038 19330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:33.627170 19244 server_base.cc:1034] running on GCE node
I20260926 14:50:33.627696 19244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:33.628729 19244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:33.629875 19244 hybrid_clock.cc:648] HybridClock initialized: now 1790434233629863 us; error 26 us; skew 500 ppm
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:33.632323 19244 init.cc:377] Logged in from keytab as oryx/127.12.68.252@KRBTEST.COM (short username oryx)
I20260926 14:50:33.633114 19244 webserver.cc:460] Webserver started at http://127.12.68.252:44679/ using document root <none> and password file <none>
I20260926 14:50:33.633574 19244 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:33.633654 19244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:33.633805 19244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:33.635607 19244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/data/instance:
uuid: "de56ca7828f84d69902976cf729cc1cb"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-24mn"
server_key: "7e99aa1737dce894291f79649df0597c"
server_key_iv: "aca0aa62a47acc53db2f9221746985c2"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.636106 19244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/wal/instance:
uuid: "de56ca7828f84d69902976cf729cc1cb"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-24mn"
server_key: "7e99aa1737dce894291f79649df0597c"
server_key_iv: "aca0aa62a47acc53db2f9221746985c2"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.639377 19244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260926 14:50:33.641620 19337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:33.642802 19244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:33.642894 19244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/wal
uuid: "de56ca7828f84d69902976cf729cc1cb"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-24mn"
server_key: "7e99aa1737dce894291f79649df0597c"
server_key_iv: "aca0aa62a47acc53db2f9221746985c2"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.642970 19244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:33.652833 19244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:33.655260 19244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:33.655390 19244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:33.662231 19244 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.252:46837
I20260926 14:50:33.662529 19389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.252:46837 every 8 connection(s)
I20260926 14:50:33.663278 19244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/data/info.pb
I20260926 14:50:33.666280 19390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:33.669721 19390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:50:33.673055 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 19244
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.252@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.68.252@KRBTEST.COM
I20260926 14:50:33.683833 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.671529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.12.68.254:43839)
0926 14:50:33.671675 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.671679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.672515 (+ 836us) server_negotiation.cc:407] Connection header received
0926 14:50:33.673667 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.673671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.673725 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.673796 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.675175 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.675928 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.677359 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.677677 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.677775 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.680372 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.680376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.680379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.680407 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.681899 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.682319 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.682322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.682324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.682421 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.682773 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.682776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.682779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.682960 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.683019 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.683570 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.683678 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:50:33.684834 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.671844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.0.0.1:55578)
0926 14:50:33.672345 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.672400 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.673413 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.674037 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.674046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.674481 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.675040 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.675053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.676057 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.676060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.677247 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.677258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.677815 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.677818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.677948 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.678601 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.678621 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.680180 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.681986 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.681994 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.682003 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.682232 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.682518 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.682521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.682523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.682677 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.683088 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.683094 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.683341 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.683514 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.683791 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":2,"spinlock_wait_cycles":2304,"thread_start_us":95,"threads_started":1}
I20260926 14:50:33.685045 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.673867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.12.68.252:46837)
0926 14:50:33.674173 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.674184 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.674205 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:50:33.674238 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.674243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.674418 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.674566 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.675411 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.676454 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.677276 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.677778 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.677946 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.680722 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.680728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.680740 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.680762 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.682659 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.683154 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.683159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.683161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.683241 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.683536 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.683539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.683541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.683815 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.684628 (+ 813us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.684732 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.684866 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":227,"thread_start_us":148,"threads_started":1}
I20260926 14:50:33.685946 19390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.252@KRBTEST.COM for oryx/127.12.68.253@KRBTEST.COM
I20260926 14:50:33.693890 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.686425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.12.68.253:44171)
0926 14:50:33.686672 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.686676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.686691 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:50:33.686730 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.686733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.686778 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.686866 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.687257 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.687920 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.688990 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.689217 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.689290 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.691156 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.691159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.691161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.691182 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.692459 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.692869 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.692871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.692872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.692937 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.693202 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.693204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.693205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.693367 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.693436 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.693689 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.693776 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":86,"threads_started":1}
I20260926 14:50:33.693923 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.686333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44171 (local address 127.0.0.1:49096)
0926 14:50:33.686463 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.686476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.686540 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.686889 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.686892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.687038 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.687136 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.687142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.688003 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.688006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.688882 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.688892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.689310 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.689314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.689412 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.689844 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.689855 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.691024 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.692552 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.692555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.692557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.692787 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.693026 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.693028 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.693030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.693113 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.693443 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.693446 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.693537 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.693646 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.693768 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 14:50:33.695407 19390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.252@KRBTEST.COM for oryx/127.12.68.252@KRBTEST.COM
I20260926 14:50:33.702589 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.695783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:46837 (local address 127.0.0.1:35224)
0926 14:50:33.695970 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.695982 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.696043 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.696228 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.696230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.696342 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.696403 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.696407 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.697340 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.697342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.697811 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.697817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.698343 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.698345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.698395 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.698785 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.698794 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.699818 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.701499 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.701501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.701503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.701667 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.701899 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.701901 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.701902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.701963 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.702276 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.702278 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.702340 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.702407 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.702487 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260926 14:50:33.702735 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.695852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.12.68.252:46837)
0926 14:50:33.695976 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.695980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.696018 (+ 38us) server_negotiation.cc:407] Connection header received
0926 14:50:33.696100 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.696103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.696143 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.696220 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.696487 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.697242 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.697900 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.698255 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.698361 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.699976 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.699979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.699982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.700006 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.701401 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.701746 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.701749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.701751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.701824 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.702026 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.702029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.702031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.702203 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.702285 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.702464 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.702570 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":56}
I20260926 14:50:33.705523 19390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: Bootstrap starting.
I20260926 14:50:33.707742 19390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:33.708461 19390 log.cc:824] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:33.710371 19390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: No bootstrap required, opened a new log
I20260926 14:50:33.714119 19390 raft_consensus.cc:348] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } }
I20260926 14:50:33.714360 19390 raft_consensus.cc:374] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:33.714402 19390 raft_consensus.cc:729] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de56ca7828f84d69902976cf729cc1cb, State: Initialized, Role: FOLLOWER
I20260926 14:50:33.714843 19390 consensus_queue.cc:260] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } }
I20260926 14:50:33.715278 19396 sys_catalog.cc:455] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } } }
I20260926 14:50:33.715399 19396 sys_catalog.cc:458] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:50:33.716158 19390 sys_catalog.cc:564] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [sys.catalog]: configured and running, proceeding with master startup.
W20260926 14:50:33.721125 19407 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:50:33.721205 19407 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:50:33.724484 19244 master_runner.cc:428] Detected that this master de56ca7828f84d69902976cf729cc1cb is joining an existing cluster
I20260926 14:50:33.724542 19244 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.68.252:46837
I20260926 14:50:33.736256 19414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.727169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:46837 (local address 127.0.0.1:35234)
0926 14:50:33.727607 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.727618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.727865 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.728185 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.728188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.728397 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.728580 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.728588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.729593 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.729596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.730567 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.730596 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.731152 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.731155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.731230 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.732234 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.732248 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.732655 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.734724 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.734727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.734729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.734982 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.735264 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.735267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.735269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.735369 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.735736 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.735738 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.735825 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.735938 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.736058 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":81,"thread_start_us":74,"threads_started":1}
I20260926 14:50:33.737919 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.727416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.12.68.252:46837)
0926 14:50:33.727505 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.727508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.727670 (+ 162us) server_negotiation.cc:407] Connection header received
0926 14:50:33.727973 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.727976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.728010 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.728050 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.728684 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.729491 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.730680 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.730959 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.731036 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.732788 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.732792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.732795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.732821 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.734602 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.735082 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.735086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.735087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.735170 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.735450 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.735454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.735456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.735651 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.735748 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.737667 (+ 1919us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.737775 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":35}
I20260926 14:50:33.738643 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 18734
I20260926 14:50:33.745675 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:38767
--local_ip_for_outbound_sockets=127.12.68.193
--tserver_master_addrs=127.12.68.254:43839
--webserver_port=43855
--webserver_interface=127.12.68.193
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.193.keytab
--principal=oryx/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.68.254:43839,127.12.68.253:44171
--tserver_master_addrs=127.12.68.254:43839,127.12.68.253:44171,127.12.68.252:46837 with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:50:33.748795 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.726849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.12.68.253:44171)
0926 14:50:33.727145 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.727149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.727488 (+ 339us) server_negotiation.cc:407] Connection header received
0926 14:50:33.727853 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.727857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.727935 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.727983 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.728671 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.729462 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.730987 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.731342 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.731415 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.745232 (+ 13817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.745237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.745239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.745279 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.747143 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.747580 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.747583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.747585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.747667 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.747922 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.747926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.747928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.748139 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.748233 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.748487 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.748646 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260926 14:50:33.748839 19415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.726751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44171 (local address 127.0.0.1:49106)
0926 14:50:33.727392 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.727405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.727736 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.728115 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.728118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.728391 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.728543 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.728551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.729596 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.729599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.730858 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.730868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.731558 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.731562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.731650 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.744450 (+ 12800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.744467 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.745102 (+ 635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.747281 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.747284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.747287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.747501 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.747747 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.747750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.747752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.747829 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.748225 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.748228 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.748335 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.748436 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.748588 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":281,"spinlock_wait_cycles":29952,"thread_start_us":80,"threads_started":1}
I20260926 14:50:33.749137 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.726947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55584 (local address 127.12.68.254:43839)
0926 14:50:33.727060 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.727065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.727216 (+ 151us) server_negotiation.cc:407] Connection header received
0926 14:50:33.727373 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.727376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.727428 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.727500 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.728331 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.729157 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.730940 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.731236 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.731308 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.745268 (+ 13960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.745272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.745275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.745305 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.747405 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.747870 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.747873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.747875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.747982 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.748272 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.748275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.748277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.748474 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.748554 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.748881 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.748995 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:50:33.749842 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.726751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.0.0.1:55584)
0926 14:50:33.727147 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.727160 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.727253 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.727835 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.727838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.728083 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.728207 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.728214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.729470 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.729473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.730824 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.730831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.731448 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.731451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.731536 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.744484 (+ 12948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.744496 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.745095 (+ 599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.747541 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.747543 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.747546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.747763 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.748096 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.748099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.748101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.748179 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.748618 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.748620 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.748709 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.749546 (+ 837us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.749684 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":344,"thread_start_us":103,"threads_started":1}
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434232, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.254@KRBTEST.COM for oryx/127.12.68.252@KRBTEST.COM
I20260926 14:50:33.761691 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.753568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:46837 (local address 127.0.0.1:35250)
0926 14:50:33.753667 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.753681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.753749 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.753993 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.753996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.754173 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.754274 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.754281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.755097 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.755101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.755939 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.755949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.756371 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.756375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.756461 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.756884 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.756899 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.758369 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.760184 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.760187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.760189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.760417 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.760672 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.760675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.760677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.760776 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.761172 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.761175 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.761286 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.761398 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.761529 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 14:50:33.761705 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.753642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.12.68.252:46837)
0926 14:50:33.753736 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.753740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.753753 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:33.753848 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.753851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.753898 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.753974 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.754379 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.754995 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.756045 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.756278 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.756362 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.758484 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.758487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.758490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.758512 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.760070 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.760504 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.760507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.760509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.760584 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.760887 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.760890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.760892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.761080 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.761177 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.761460 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.761566 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:50:33.762534 18672 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.68.252:46837: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "de56ca7828f84d69902976cf729cc1cb"
member_type: NON_VOTER
last_known_addr {
host: "127.12.68.252"
port: 46837
}
attrs {
promote: true
}
}
dest_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a"
cas_config_opid_index: 5
I20260926 14:50:33.763146 18672 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: NON_VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: true } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } }
I20260926 14:50:33.764166 19230 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER de56ca7828f84d69902976cf729cc1cb (127.12.68.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: NON_VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: true } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } } }
I20260926 14:50:33.765652 19230 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.68.252:46837
I20260926 14:50:33.765832 19231 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: NON_VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: true } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } } }
I20260926 14:50:33.765933 19231 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:33.766218 19244 master.cc:537] Master@127.12.68.252:46837 shutting down...
I20260926 14:50:33.766546 19364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:33.767922 19364 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 1 FOLLOWER]: Refusing update from remote peer ccf70cbde5ec4195a72786c4dd8b3e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 14:50:33.768631 19231 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 14:50:33.770891 18656 consensus_peers.cc:487] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a -> Peer de56ca7828f84d69902976cf729cc1cb (127.12.68.252:46837): Couldn't send request to peer de56ca7828f84d69902976cf729cc1cb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 14:50:33.771598 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.764212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44171 (local address 127.0.0.1:49108)
0926 14:50:33.766729 (+ 2517us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.766743 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.766821 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.767176 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.767179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.767194 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:33.767407 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.767414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.768451 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.768454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.769854 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.769866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.771198 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.771202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.771296 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.771341 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.771416 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.771480 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2385}
I20260926 14:50:33.771608 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.764471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.12.68.253:44171)
0926 14:50:33.764574 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.764578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.766892 (+ 2314us) server_negotiation.cc:407] Connection header received
0926 14:50:33.766939 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.766942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.766996 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.767070 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:33.767521 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.768329 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.769988 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.770874 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.770976 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.771127 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.771436 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.771515 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:50:33.772302 19282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 LEARNER]: Refusing update from remote peer ccf70cbde5ec4195a72786c4dd8b3e5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 14:50:33.772778 19231 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: NON_VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 14:50:33.773402 19244 raft_consensus.cc:2229] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:33.773706 19244 raft_consensus.cc:2258] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:33.773852 19244 tablet_replica.cc:330] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: stopping tablet replica
I20260926 14:50:33.774626 19282 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER de56ca7828f84d69902976cf729cc1cb (127.12.68.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: NON_VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: true } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } } }
I20260926 14:50:33.776664 19418 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: NON_VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: true } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } } }
I20260926 14:50:33.776840 19418 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:50:33.776867 19311 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccf70cbde5ec4195a72786c4dd8b3e5a. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: NON_VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: true } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } } }
I20260926 14:50:33.776947 19311 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:50:33.777900 19231 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a: attempting to promote NON_VOTER e7d0f6987f9e42e586b68f89d0bc95b4 to VOTER
I20260926 14:50:33.778319 19231 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } }
I20260926 14:50:33.779480 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.779231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:46837 (local address 127.0.0.1:35256)
0926 14:50:33.779338 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.779409 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:46837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 14:50:33.779606 19282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 LEARNER]: Refusing update from remote peer ccf70cbde5ec4195a72786c4dd8b3e5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 14:50:33.780037 19230 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 14:50:33.780534 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.780356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:46837 (local address 127.0.0.1:35266)
0926 14:50:33.780437 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.780476 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:46837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 14:50:33.780743 18656 consensus_peers.cc:487] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a -> Peer de56ca7828f84d69902976cf729cc1cb (127.12.68.252:46837): Couldn't send request to peer de56ca7828f84d69902976cf729cc1cb. Status: Network error: Client connection negotiation failed: client connection to 127.12.68.252:46837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 14:50:33.783299 19231 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e7d0f6987f9e42e586b68f89d0bc95b4 (127.12.68.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } } }
I20260926 14:50:33.784970 19281 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e7d0f6987f9e42e586b68f89d0bc95b4 (127.12.68.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } } }
I20260926 14:50:33.786352 19311 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } } }
I20260926 14:50:33.786464 19311 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:50:33.786808 19230 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } } }
I20260926 14:50:33.786912 19230 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:33.786897 19231 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } } }
I20260926 14:50:33.787122 19231 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:33.788668 19244 master.cc:552] Master@127.12.68.252:46837 shutdown complete.
I20260926 14:50:33.788718 19244 master_runner.cc:305] Clearing existing system tablet
I20260926 14:50:33.788954 19244 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:33.788996 19244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:33.790311 19244 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:33.791164 19422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:33.791363 19244 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:33.791591 19244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:33.791678 19244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/wal
uuid: "de56ca7828f84d69902976cf729cc1cb"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-24mn"
server_key: "7e99aa1737dce894291f79649df0597c"
server_key_iv: "aca0aa62a47acc53db2f9221746985c2"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.793900 19244 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:33.796330 19244 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 14:50:33.796486 19244 log.cc:1196] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 14:50:33.796816 19244 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: Deleting consensus metadata
I20260926 14:50:33.797015 19244 master_runner.cc:315] Copying system tablet from 127.12.68.254:43839
I20260926 14:50:33.798187 19244 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: tablet copy: Beginning tablet copy session from remote peer at address 127.12.68.254:43839
I20260926 14:50:33.807099 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.798723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.0.0.1:55598)
0926 14:50:33.799028 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.799041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.799178 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.799515 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.799518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.799726 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.799848 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.799854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.800940 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.800944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.801878 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.801890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.803290 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.803294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.803394 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.803993 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.804007 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.804309 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.805775 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.805778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.805780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.805962 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.806203 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.806206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.806208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.806283 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.806607 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.806618 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.806703 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.806786 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.806910 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":79,"threads_started":1}
I20260926 14:50:33.807089 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.798788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55598 (local address 127.12.68.254:43839)
0926 14:50:33.798898 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.798903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.799107 (+ 204us) server_negotiation.cc:407] Connection header received
0926 14:50:33.799284 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.799287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.799333 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.799380 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.799988 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.800796 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.802017 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.802377 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.802457 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.804440 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.804444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.804446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.804470 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.805680 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.806045 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.806048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.806050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.806126 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.806357 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.806360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.806362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.806527 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.806606 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.806844 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.806938 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:50:33.807763 18692 tablet_copy_service.cc:140] P ccf70cbde5ec4195a72786c4dd8b3e5a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer de56ca7828f84d69902976cf729cc1cb ({username='oryx', principal='oryx/127.12.68.252@KRBTEST.COM'} at 127.0.0.1:55598)
I20260926 14:50:33.807843 18692 tablet_copy_service.cc:161] P ccf70cbde5ec4195a72786c4dd8b3e5a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer de56ca7828f84d69902976cf729cc1cb at {username='oryx', principal='oryx/127.12.68.252@KRBTEST.COM'} at 127.0.0.1:55598: session id = de56ca7828f84d69902976cf729cc1cb-00000000000000000000000000000000
I20260926 14:50:33.808733 18692 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 14:50:33.809721 19244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 14:50:33.811286 19244 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: tablet copy: Starting download of 0 data blocks...
I20260926 14:50:33.811434 19244 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: tablet copy: Starting download of 1 WAL segments...
I20260926 14:50:33.812644 19244 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 14:50:33.813913 18692 tablet_copy_service.cc:342] P ccf70cbde5ec4195a72786c4dd8b3e5a: Request end of tablet copy session de56ca7828f84d69902976cf729cc1cb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.68.252@KRBTEST.COM'} at 127.0.0.1:55598
I20260926 14:50:33.813977 18692 tablet_copy_service.cc:433] P ccf70cbde5ec4195a72786c4dd8b3e5a: ending tablet copy session de56ca7828f84d69902976cf729cc1cb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer de56ca7828f84d69902976cf729cc1cb
I20260926 14:50:33.819550 19244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:33.820993 19429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:33.821058 19432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:33.821269 19244 server_base.cc:1034] running on GCE node
W20260926 14:50:33.824007 19430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:33.824235 19244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:33.824498 19244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:33.825646 19244 hybrid_clock.cc:648] HybridClock initialized: now 1790434233825632 us; error 28 us; skew 500 ppm
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:33.826856 19244 init.cc:377] Logged in from keytab as oryx/127.12.68.252@KRBTEST.COM (short username oryx)
I20260926 14:50:33.827246 19244 webserver.cc:460] Webserver started at http://127.12.68.252:39885/ using document root <none> and password file <none>
I20260926 14:50:33.827414 19244 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:33.827446 19244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:33.828466 19244 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:33.829250 19439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:33.829452 19244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:33.829519 19244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/wal
uuid: "de56ca7828f84d69902976cf729cc1cb"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-24mn"
server_key: "7e99aa1737dce894291f79649df0597c"
server_key_iv: "aca0aa62a47acc53db2f9221746985c2"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.829594 19244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:33.843520 19244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:33.845157 19244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:33.849418 19244 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.252:46837
I20260926 14:50:33.849433 19491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.252:46837 every 8 connection(s)
I20260926 14:50:33.849920 19244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/master-2/data/info.pb
I20260926 14:50:33.857123 19492 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:50:33.857937 19492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: Bootstrap starting.
I20260926 14:50:33.866199 19492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:50:33.866446 19492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: Bootstrap complete.
I20260926 14:50:33.866892 19492 raft_consensus.cc:348] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } }
I20260926 14:50:33.867007 19492 raft_consensus.cc:729] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de56ca7828f84d69902976cf729cc1cb, State: Initialized, Role: LEARNER
I20260926 14:50:33.867101 19492 consensus_queue.cc:260] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } }
I20260926 14:50:33.867401 19495 sys_catalog.cc:455] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } } }
I20260926 14:50:33.867538 19492 sys_catalog.cc:564] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:33.867563 19495 sys_catalog.cc:458] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [sys.catalog]: This master's current role is: LEARNER
I20260926 14:50:33.870040 19506 catalog_manager.cc:1259] Loaded cluster ID: 71d27facd073479e96c0cc5b21759065
I20260926 14:50:33.870142 19506 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: loading cluster ID for follower catalog manager: success
I20260926 14:50:33.871953 19506 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: acquiring CA information for follower catalog manager: success
I20260926 14:50:33.872991 19506 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 14:50:33.918643 19416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:33.918900 19416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:33.918960 19416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:33.922922 19416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:33.922978 19416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:33.923058 19416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:33.928196 19416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.193.keytab
--principal=oryx/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:38767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=43855
--tserver_master_addrs=127.12.68.254:43839,127.12.68.253:44171,127.12.68.252:46837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:33.929183 19416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:33.930158 19416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:33.939208 19512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:33.939278 19416 server_base.cc:1034] running on GCE node
W20260926 14:50:33.939399 19511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:33.939241 19514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:33.939899 19416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:33.940464 19416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:33.941628 19416 hybrid_clock.cc:648] HybridClock initialized: now 1790434233941616 us; error 37 us; skew 500 ppm
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:33.944130 19416 init.cc:377] Logged in from keytab as oryx/127.12.68.193@KRBTEST.COM (short username oryx)
I20260926 14:50:33.944897 19416 webserver.cc:460] Webserver started at http://127.12.68.193:43855/ using document root <none> and password file <none>
I20260926 14:50:33.945384 19416 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:33.945451 19416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:33.949026 19416 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:33.951053 19521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:33.952353 19416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:33.952454 19416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "9c41ef8146da478c86767ede40d05247"
format_stamp: "Formatted at 2026-09-26 14:50:32 on dist-test-slave-24mn"
server_key: "97645633095fd9768184af480221e48f"
server_key_iv: "6b752c71e48e29c55984fb8ebe09686e"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.952960 19416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:33.964778 19416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:33.967622 19416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:33.967770 19416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:33.968405 19416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:33.969230 19416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:33.969275 19416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:33.969311 19416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:33.969331 19416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:33.977681 19416 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:38767
I20260926 14:50:33.977741 19634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:38767 every 8 connection(s)
I20260926 14:50:33.978626 19416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:33.981010 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 19416
I20260926 14:50:33.981196 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 18868
I20260926 14:50:33.987762 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:45333
--local_ip_for_outbound_sockets=127.12.68.194
--tserver_master_addrs=127.12.68.254:43839
--webserver_port=44259
--webserver_interface=127.12.68.194
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.194.keytab
--principal=oryx/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.68.254:43839,127.12.68.253:44171
--tserver_master_addrs=127.12.68.254:43839,127.12.68.253:44171,127.12.68.252:46837 with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.253@KRBTEST.COM
Sep 26 14:50:33 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.252@KRBTEST.COM
I20260926 14:50:33.996263 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.982221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:37201 (local address 127.12.68.254:43839)
0926 14:50:33.982359 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.982364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.982741 (+ 377us) server_negotiation.cc:407] Connection header received
0926 14:50:33.983933 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.983936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.983994 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.984066 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.985812 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.986590 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.988063 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.988380 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.988446 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.991795 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.991798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.991799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.991824 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.993216 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.993697 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.993701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.993702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.993787 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.994154 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.994157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.994159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.994386 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.994476 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.995990 (+ 1514us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.996112 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:50:33.996430 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.982863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:57239 (local address 127.12.68.253:44171)
0926 14:50:33.982977 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.982981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.983198 (+ 217us) server_negotiation.cc:407] Connection header received
0926 14:50:33.984060 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.984064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.984118 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.984192 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.986740 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.987424 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.988854 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.989177 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.989239 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.992860 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.992863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.992866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.992893 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.994819 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.995248 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.995252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.995253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.995338 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.995600 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.995603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.995605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.995796 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.995866 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.996200 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.996305 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:50:33.997316 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.980842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.193:37201)
0926 14:50:33.982584 (+ 1742us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.982648 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.983698 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.984347 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.984356 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.984895 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.985658 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.985678 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.986745 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.986747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.987943 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.987954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.988649 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.988652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.988838 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.989461 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.989478 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.991650 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.993304 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.993309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.993324 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.993611 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.993890 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.993893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.993894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.994053 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.994527 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.994534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.994841 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.995035 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.995356 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1535,"mutex_wait_us":78,"spinlock_wait_cycles":12288,"thread_start_us":572,"threads_started":1}
I20260926 14:50:33.997712 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.980584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:50345 (local address 127.12.68.252:46837)
0926 14:50:33.987488 (+ 6904us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.987501 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.987514 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:33.987551 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.987553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.987602 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.987677 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.988251 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.989286 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.990301 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.990603 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.990691 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.993886 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.993890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.993893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.993920 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.995744 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.996228 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.996232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.996234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.996316 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.996613 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.996616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.996618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.996821 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.996907 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.997418 (+ 511us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.997535 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6823,"thread_start_us":94,"threads_started":1}
I20260926 14:50:33.997574 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.982603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44171 (local address 127.12.68.193:57239)
0926 14:50:33.983095 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.983117 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.983970 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.984347 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.984356 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.984895 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.985658 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.985678 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.987545 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.987548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.988735 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.988744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.989382 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.989386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.989537 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.989985 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.989998 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.992706 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.994927 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.994930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.994932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.995152 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.995435 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.995437 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.995439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.995513 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.995940 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.995943 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.996019 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.996589 (+ 570us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.996709 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":117,"spinlock_wait_cycles":2304,"thread_start_us":79,"threads_started":1}
I20260926 14:50:33.998351 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.982202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:46837 (local address 127.12.68.193:50345)
0926 14:50:33.982840 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:33.982853 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:33.983694 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:33.987787 (+ 4093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:33.987790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:33.987986 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:33.988119 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.988125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.989390 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.989392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.990176 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:33.990186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.990763 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.990769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:33.990852 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.991218 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:33.991227 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:33.993735 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:33.995866 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.995870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.995872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.996119 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.996413 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:33.996416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:33.996418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:33.996502 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:33.996951 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:33.996954 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:33.997040 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 14:50:33.997141 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 14:50:33.997266 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":25,"spinlock_wait_cycles":24320,"thread_start_us":114,"threads_started":1}
I20260926 14:50:33.999127 19636 heartbeater.cc:347] Connected to a master server at 127.12.68.254:43839
I20260926 14:50:33.999348 19636 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:33.999637 19635 heartbeater.cc:347] Connected to a master server at 127.12.68.252:46837
I20260926 14:50:33.999711 19635 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:33.999841 19636 heartbeater.cc:511] Master 127.12.68.254:43839 requested a full tablet report, sending...
I20260926 14:50:33.999851 19635 heartbeater.cc:511] Master 127.12.68.252:46837 requested a full tablet report, sending...
I20260926 14:50:33.999951 19637 heartbeater.cc:347] Connected to a master server at 127.12.68.253:44171
I20260926 14:50:34.000038 19637 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:34.000190 19637 heartbeater.cc:511] Master 127.12.68.253:44171 requested a full tablet report, sending...
I20260926 14:50:34.001150 18672 ts_manager.cc:194] Re-registered known tserver with Master: 9c41ef8146da478c86767ede40d05247 (127.12.68.193:38767)
I20260926 14:50:34.001339 19456 ts_manager.cc:194] Registered new tserver with Master: 9c41ef8146da478c86767ede40d05247 (127.12.68.193:38767)
I20260926 14:50:34.001348 19264 ts_manager.cc:194] Registered new tserver with Master: 9c41ef8146da478c86767ede40d05247 (127.12.68.193:38767)
I20260926 14:50:34.001866 18672 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:37201
W20260926 14:50:34.099124 19643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:34.099359 19643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:34.099398 19643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:34.103219 19643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:34.103266 19643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:34.103322 19643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:34.108393 19643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.194.keytab
--principal=oryx/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:45333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=44259
--tserver_master_addrs=127.12.68.254:43839,127.12.68.253:44171,127.12.68.252:46837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:34.109337 19643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:34.110240 19643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:34.119349 19649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:34.119323 19651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:34.119349 19648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:34.119649 19643 server_base.cc:1034] running on GCE node
I20260926 14:50:34.120061 19643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:34.120654 19643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:34.121803 19643 hybrid_clock.cc:648] HybridClock initialized: now 1790434234121785 us; error 33 us; skew 500 ppm
Sep 26 14:50:34 dist-test-slave-24mn krb5kdc[18627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:34.124293 19643 init.cc:377] Logged in from keytab as oryx/127.12.68.194@KRBTEST.COM (short username oryx)
I20260926 14:50:34.125113 19643 webserver.cc:460] Webserver started at http://127.12.68.194:44259/ using document root <none> and password file <none>
I20260926 14:50:34.125622 19643 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:34.125700 19643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:34.129333 19643 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:34.131299 19658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.132557 19643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:34.132663 19643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "f7b60ac486fb400e95bb50df4ed3eb98"
format_stamp: "Formatted at 2026-09-26 14:50:32 on dist-test-slave-24mn"
server_key: "2b529ce575fcff41f21d9bc69121d997"
server_key_iv: "cc4d2be2d2352c688aef8e89979a146f"
server_key_version: "encryptionkey@0"
I20260926 14:50:34.133199 19643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:34.149096 19643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:34.151577 19643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:34.151773 19643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:34.152324 19643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:34.153107 19643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:34.153152 19643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.153199 19643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:34.153227 19643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.161876 19643 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:45333
I20260926 14:50:34.161918 19771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:45333 every 8 connection(s)
I20260926 14:50:34.162799 19643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:34.163123 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.159535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:46837 (local address 127.0.0.1:35272)
0926 14:50:34.159638 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.159653 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.159739 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.160245 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.160248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.160269 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:34.160368 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.160375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.161149 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.161153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.162057 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.162069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.162735 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.162738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.162821 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.162843 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.162916 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.162987 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 14:50:34.163182 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.159627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.12.68.252:46837)
0926 14:50:34.159869 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.159873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.159908 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:50:34.159959 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.159962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.160022 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.160122 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:34.160477 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.161044 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.162181 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.162619 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.162712 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.162855 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.162965 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.163056 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260926 14:50:34.163733 19466 raft_consensus.cc:1203] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 14:50:34.164263 19495 sys_catalog.cc:455] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccf70cbde5ec4195a72786c4dd8b3e5a. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } } }
I20260926 14:50:34.164392 19495 sys_catalog.cc:458] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [sys.catalog]: This master's current role is: LEARNER
I20260926 14:50:34.165202 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 19643
I20260926 14:50:34.165385 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 19002
I20260926 14:50:34.172560 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:44785
--local_ip_for_outbound_sockets=127.12.68.195
--tserver_master_addrs=127.12.68.254:43839
--webserver_port=36097
--webserver_interface=127.12.68.195
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.195.keytab
--principal=oryx/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.68.254:43839,127.12.68.253:44171
--tserver_master_addrs=127.12.68.254:43839,127.12.68.253:44171,127.12.68.252:46837 with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:50:34 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.253@KRBTEST.COM
Sep 26 14:50:34 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
Sep 26 14:50:34 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.252@KRBTEST.COM
I20260926 14:50:34.179678 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.166273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:41287 (local address 127.12.68.252:46837)
0926 14:50:34.170537 (+ 4264us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.170541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.170554 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:34.170586 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.170589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.170628 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.170702 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.171162 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.171840 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.173338 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.173641 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.173739 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.175715 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.175718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.175721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.175744 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.177194 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.177838 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.177841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.177843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.177928 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.178388 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.178392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.178394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.178653 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.178712 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.179399 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.179517 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4189}
I20260926 14:50:34.179678 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.166409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:38925 (local address 127.12.68.253:44171)
0926 14:50:34.166514 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.166518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.167149 (+ 631us) server_negotiation.cc:407] Connection header received
0926 14:50:34.167717 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.167720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.167800 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.167854 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.169209 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.169941 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.171092 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.171397 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.171465 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.175840 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.175844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.175847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.175874 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.177335 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.177838 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.177841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.177843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.178016 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.178388 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.178392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.178394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.178600 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.178676 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.179399 (+ 723us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.179517 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:50:34.179764 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.165177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:36135 (local address 127.12.68.254:43839)
0926 14:50:34.165290 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.165295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.166771 (+ 1476us) server_negotiation.cc:407] Connection header received
0926 14:50:34.167701 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.167704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.167752 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.167800 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.169179 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.169764 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.170989 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.171283 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.171348 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.175504 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.175508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.175511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.175543 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.177838 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.178313 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.178316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.178318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.178405 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.178705 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.178708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.178710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.178930 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.179001 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.179478 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.179595 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:50:34.181306 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.165020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.194:36135)
0926 14:50:34.166682 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.166692 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.167484 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.168076 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.168083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.168552 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:34.169056 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.169068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.169899 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.169902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.170865 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.170873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.171533 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.171536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.171717 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.173738 (+ 2021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:34.173753 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:34.175347 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:34.177963 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.177967 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.177969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.178196 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.178517 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.178520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.178522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.178605 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.179074 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:34.179077 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:34.179176 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.180362 (+ 1186us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.180553 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1589,"mutex_wait_us":63,"spinlock_wait_cycles":56064,"thread_start_us":125,"threads_started":1}
I20260926 14:50:34.181315 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.165019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:46837 (local address 127.12.68.194:41287)
0926 14:50:34.166553 (+ 1534us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.166604 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.167482 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.170736 (+ 3254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.170738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.170950 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:34.171046 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.171051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.172458 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.172461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.173223 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.173233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.173880 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.173884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.173958 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.174363 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:34.174374 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:34.175570 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:34.177327 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.177333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.177347 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.177731 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.178125 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.178127 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.178129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.178285 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.178807 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:34.178809 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:34.179028 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.179822 (+ 794us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.180214 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"mutex_wait_us":10,"spinlock_wait_cycles":21504,"thread_start_us":129,"threads_started":1}
I20260926 14:50:34.182847 19772 heartbeater.cc:347] Connected to a master server at 127.12.68.252:46837
I20260926 14:50:34.182963 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.166006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44171 (local address 127.12.68.194:38925)
0926 14:50:34.167057 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.167069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.167489 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.168078 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.168084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.168564 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:34.169059 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.169072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.170074 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.170077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.170979 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.170989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.171639 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.171642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.171752 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.172387 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:34.172408 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:34.175710 (+ 3302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:34.177432 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.177434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.177436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.177731 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.178102 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.178105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.178107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.178285 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.178733 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:34.178739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:34.179028 (+ 289us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.179937 (+ 909us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.180270 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":177,"spinlock_wait_cycles":21248,"thread_start_us":73,"threads_started":1}
I20260926 14:50:34.183091 19772 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:34.183465 19774 heartbeater.cc:347] Connected to a master server at 127.12.68.253:44171
I20260926 14:50:34.183537 19774 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:34.183626 19772 heartbeater.cc:511] Master 127.12.68.252:46837 requested a full tablet report, sending...
I20260926 14:50:34.183673 19774 heartbeater.cc:511] Master 127.12.68.253:44171 requested a full tablet report, sending...
I20260926 14:50:34.182859 19773 heartbeater.cc:347] Connected to a master server at 127.12.68.254:43839
I20260926 14:50:34.184286 19773 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:34.184422 19773 heartbeater.cc:511] Master 127.12.68.254:43839 requested a full tablet report, sending...
I20260926 14:50:34.184831 19456 ts_manager.cc:194] Registered new tserver with Master: f7b60ac486fb400e95bb50df4ed3eb98 (127.12.68.194:45333)
I20260926 14:50:34.185149 19264 ts_manager.cc:194] Registered new tserver with Master: f7b60ac486fb400e95bb50df4ed3eb98 (127.12.68.194:45333)
I20260926 14:50:34.185427 18672 ts_manager.cc:194] Re-registered known tserver with Master: f7b60ac486fb400e95bb50df4ed3eb98 (127.12.68.194:45333)
I20260926 14:50:34.186133 18672 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:36135
W20260926 14:50:34.280225 19779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:34.280447 19779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:34.280488 19779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:34.284420 19779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:34.284469 19779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:34.284523 19779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:34.289535 19779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.195.keytab
--principal=oryx/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:44785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=36097
--tserver_master_addrs=127.12.68.254:43839,127.12.68.253:44171,127.12.68.252:46837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:34.290400 19779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:34.291263 19779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:34.300119 19785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:34.300114 19784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:34.300186 19779 server_base.cc:1034] running on GCE node
W20260926 14:50:34.300143 19787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:34.300644 19779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:34.301193 19779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:34.302336 19779 hybrid_clock.cc:648] HybridClock initialized: now 1790434234302311 us; error 35 us; skew 500 ppm
Sep 26 14:50:34 dist-test-slave-24mn krb5kdc[18627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:34.304819 19779 init.cc:377] Logged in from keytab as oryx/127.12.68.195@KRBTEST.COM (short username oryx)
I20260926 14:50:34.305550 19779 webserver.cc:460] Webserver started at http://127.12.68.195:36097/ using document root <none> and password file <none>
I20260926 14:50:34.306015 19779 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:34.306078 19779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:34.309597 19779 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:34.311359 19794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.312494 19779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:34.312589 19779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "687fc8c182854d2da4f7d2a7418ce476"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-24mn"
server_key: "4985b000d1ac09645d249b6fad13d596"
server_key_iv: "4cd716d4f3143d5d21ce4c3a808be058"
server_key_version: "encryptionkey@0"
I20260926 14:50:34.313067 19779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:34.327507 19779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:34.330106 19779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:34.330255 19779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:34.330834 19779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:34.331620 19779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:34.331669 19779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.331764 19779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:34.331796 19779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.339938 19779 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:44785
I20260926 14:50:34.339978 19907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:44785 every 8 connection(s)
I20260926 14:50:34.340832 19779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:34.348605 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 19779
Sep 26 14:50:34 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
Sep 26 14:50:34 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.253@KRBTEST.COM
Sep 26 14:50:34 dist-test-slave-24mn krb5kdc[18627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.252@KRBTEST.COM
I20260926 14:50:34.355793 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.342490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:52205 (local address 127.12.68.254:43839)
0926 14:50:34.342622 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.342626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.343747 (+ 1121us) server_negotiation.cc:407] Connection header received
0926 14:50:34.344845 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.344848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.344891 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.344987 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.346412 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.347196 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.348874 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.349171 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.349244 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.351767 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.351771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.351774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.351801 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.353553 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.354135 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.354139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.354141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.354225 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.354602 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.354606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.354608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.354819 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.354884 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.355529 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.355636 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:50:34.357393 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.342840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.195:52205)
0926 14:50:34.343614 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.343652 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.344636 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.345178 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.345186 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.345678 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:34.346243 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.346256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.347974 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.347977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.348744 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.348754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.349391 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.349395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.349488 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.350001 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:34.350021 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:34.351619 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:34.353674 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.353680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.353695 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.354007 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.354336 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.354339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.354341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.354495 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.354961 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:34.354967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:34.355260 (+ 293us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.356018 (+ 758us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.356336 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":110,"thread_start_us":486,"threads_started":1}
I20260926 14:50:34.357529 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.344372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44171 (local address 127.12.68.195:39801)
0926 14:50:34.345040 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.345051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.345131 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.345344 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.345347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.345673 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:34.346241 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.346254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.347322 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.347325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.348470 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.348481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.349201 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.349204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.349375 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.350197 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:34.350211 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:34.352345 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:34.355476 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.355479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.355480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.355696 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.356290 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.356293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.356295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.356378 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.357020 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:34.357023 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:34.357126 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.357219 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.357335 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":100,"spinlock_wait_cycles":11008,"thread_start_us":63,"threads_started":1}
I20260926 14:50:34.358697 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.344454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:39801 (local address 127.12.68.253:44171)
0926 14:50:34.344553 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.344557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.345109 (+ 552us) server_negotiation.cc:407] Connection header received
0926 14:50:34.345205 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.345207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.345244 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.345307 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.346370 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.347180 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.348617 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.348953 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.349033 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.352479 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.352483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.352485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.352506 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.355374 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.355794 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.355798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.355800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.355900 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.356476 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.356479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.356481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.356685 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.356768 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.358442 (+ 1674us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.358537 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 14:50:34.358774 19909 heartbeater.cc:347] Connected to a master server at 127.12.68.254:43839
I20260926 14:50:34.358986 19909 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:34.359200 19910 heartbeater.cc:347] Connected to a master server at 127.12.68.253:44171
I20260926 14:50:34.359274 19910 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:34.359644 19910 heartbeater.cc:511] Master 127.12.68.253:44171 requested a full tablet report, sending...
I20260926 14:50:34.360761 19264 ts_manager.cc:194] Registered new tserver with Master: 687fc8c182854d2da4f7d2a7418ce476 (127.12.68.195:44785)
I20260926 14:50:34.361022 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.344106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:34431 (local address 127.12.68.252:46837)
0926 14:50:34.344209 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.344214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.344711 (+ 497us) server_negotiation.cc:407] Connection header received
0926 14:50:34.344838 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.344841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.344891 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.344987 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.347316 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.347804 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.349082 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.349399 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.349469 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.352837 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.352841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.352844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.352870 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.354696 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.359826 (+ 5130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.359830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.359832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.359932 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.360205 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.360208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.360210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.360394 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.360462 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.360772 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.360882 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:50:34.361541 19909 heartbeater.cc:511] Master 127.12.68.254:43839 requested a full tablet report, sending...
I20260926 14:50:34.361935 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.350617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.12.68.253:44171)
0926 14:50:34.352971 (+ 2354us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.352974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.352983 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:34.353012 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.353014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.353047 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.353127 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.354165 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.354905 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.356019 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.356369 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.356456 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.357956 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.357959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.357962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.357985 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.359529 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.359996 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.359999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.360001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.360073 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.360361 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.360364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.360366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.360540 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.360646 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.361691 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.361796 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2298,"thread_start_us":77,"threads_started":1}
I20260926 14:50:34.362155 18672 ts_manager.cc:194] Re-registered known tserver with Master: 687fc8c182854d2da4f7d2a7418ce476 (127.12.68.195:44785)
I20260926 14:50:34.362210 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.344010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:46837 (local address 127.12.68.195:34431)
0926 14:50:34.344633 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.344645 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.344739 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.345251 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.345254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.345686 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:34.346248 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.346256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.347934 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.347938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.348967 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.348976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.349623 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.349626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.349711 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.350091 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:34.350105 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:34.352718 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:34.359430 (+ 6712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.359433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.359435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.359671 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.360034 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.360037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.360039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.360111 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.360515 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:34.360517 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:34.360591 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.361346 (+ 755us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.361478 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":132,"spinlock_wait_cycles":13184,"thread_start_us":116,"threads_started":1}
I20260926 14:50:34.362712 19908 heartbeater.cc:347] Connected to a master server at 127.12.68.252:46837
I20260926 14:50:34.362804 19908 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:34.362855 18672 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:52205
I20260926 14:50:34.362948 19908 heartbeater.cc:511] Master 127.12.68.252:46837 requested a full tablet report, sending...
I20260926 14:50:34.363524 19456 ts_manager.cc:194] Registered new tserver with Master: 687fc8c182854d2da4f7d2a7418ce476 (127.12.68.195:44785)
I20260926 14:50:34.366586 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.351641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.12.68.252:46837)
0926 14:50:34.353306 (+ 1665us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.353319 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.354196 (+ 877us) server_negotiation.cc:407] Connection header received
0926 14:50:34.354231 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.354234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.354276 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.354347 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.355853 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.356596 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.359414 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.359721 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.359822 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.363141 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.363145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.363147 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.363168 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.365143 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.365515 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.365518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.365520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.365594 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.365841 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.365843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.365844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.366011 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.366129 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.366317 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.366432 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1598,"spinlock_wait_cycles":1408,"thread_start_us":60,"threads_started":1}
I20260926 14:50:34.367139 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.351069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.12.68.254:43839)
0926 14:50:34.351836 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.351840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.353673 (+ 1833us) server_negotiation.cc:407] Connection header received
0926 14:50:34.353716 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.353720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.353782 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.353859 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.356709 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.357473 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.359952 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.360266 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.360359 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.364388 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.364391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.364394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.364418 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.365636 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.366021 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.366024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.366026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.366120 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.366354 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.366357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.366358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.366542 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.366666 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.366878 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.366989 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":53,"threads_started":1}
I20260926 14:50:34.369701 18672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:34.372798 18672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:34.375047 19419 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: NON_VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: true } } }
I20260926 14:50:34.375166 19419 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:34.378209 19419 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a: attempting to promote NON_VOTER de56ca7828f84d69902976cf729cc1cb to VOTER
I20260926 14:50:34.378525 19419 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } }
I20260926 14:50:34.380263 19466 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 1 LEARNER]: Refusing update from remote peer ccf70cbde5ec4195a72786c4dd8b3e5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 14:50:34.380313 19281 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 FOLLOWER]: Refusing update from remote peer ccf70cbde5ec4195a72786c4dd8b3e5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 14:50:34.380635 19927 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 14:50:34.380882 19926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 14:50:34.382259 19928 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, de56ca7828f84d69902976cf729cc1cb (127.12.68.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } } }
I20260926 14:50:34.382879 19281 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, de56ca7828f84d69902976cf729cc1cb (127.12.68.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } } }
I20260926 14:50:34.382982 19466 raft_consensus.cc:2932] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, de56ca7828f84d69902976cf729cc1cb (127.12.68.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } } }
I20260926 14:50:34.383654 19928 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } } }
I20260926 14:50:34.383744 19928 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:34.383858 19928 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } } }
I20260926 14:50:34.383971 19928 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:34.384085 19928 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } } }
I20260926 14:50:34.384164 19928 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:34.384752 19495 sys_catalog.cc:455] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } } }
I20260926 14:50:34.384769 19936 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } } }
I20260926 14:50:34.384843 19495 sys_catalog.cc:458] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:50:34.384876 19936 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:50:34.392501 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.387309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:45333 (local address 127.0.0.1:60174)
0926 14:50:34.387449 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.387462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.387527 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.389022 (+ 1495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.389024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.389040 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:34.389115 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.389119 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.390457 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.390460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.391352 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.391361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.392161 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.392164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.392247 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.392267 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.392322 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.392386 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 14:50:34.392550 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.387687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.12.68.194:45333)
0926 14:50:34.388545 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.388558 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.388582 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:50:34.388660 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.388666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.388889 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.389012 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:34.389233 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.390356 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.391458 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.392058 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.392153 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.392303 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.392362 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.392442 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":188,"threads_started":1}
I20260926 14:50:34.394227 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.387840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:44785 (local address 127.0.0.1:38814)
0926 14:50:34.388701 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.388713 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.388788 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.389617 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.389619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.389630 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:34.389693 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.389696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.391531 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.391534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.392765 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.392775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.393871 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.393875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.393958 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.393980 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.394043 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.394108 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":95,"threads_started":1}
I20260926 14:50:34.394593 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.388213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.12.68.195:44785)
0926 14:50:34.389240 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.389250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.389270 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:50:34.389334 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.389339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.389505 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.389648 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:34.389766 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.390795 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.392888 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.393555 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.393673 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.393810 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.394382 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.394471 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":937,"thread_start_us":87,"threads_started":1}
I20260926 14:50:34.395665 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.387557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:38767 (local address 127.0.0.1:36630)
0926 14:50:34.387794 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.387807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.387923 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.391387 (+ 3464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.391390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.391406 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:34.391487 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.391493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.393053 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.393056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.394544 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.394553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.395295 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.395298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.395388 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.395409 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.395482 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.395545 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
I20260926 14:50:34.395723 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.387678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.12.68.193:38767)
0926 14:50:34.391060 (+ 3382us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.391070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.391090 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:50:34.391147 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.391150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.391270 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.391390 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:34.391601 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.392948 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.394651 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.395186 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.395283 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.395411 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.395507 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.395592 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3281,"thread_start_us":92,"threads_started":1}
I20260926 14:50:34.395532 19706 tablet_service.cc:1467] Processing CreateTablet for tablet a96a74aab7e64898a040f093333f500b (DEFAULT_TABLE table=test-table [id=135db6f079884f038f42bed56d400d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:34.396473 19706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96a74aab7e64898a040f093333f500b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:34.397552 19842 tablet_service.cc:1467] Processing CreateTablet for tablet a96a74aab7e64898a040f093333f500b (DEFAULT_TABLE table=test-table [id=135db6f079884f038f42bed56d400d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:34.398403 19842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96a74aab7e64898a040f093333f500b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:34.398285 19569 tablet_service.cc:1467] Processing CreateTablet for tablet a96a74aab7e64898a040f093333f500b (DEFAULT_TABLE table=test-table [id=135db6f079884f038f42bed56d400d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:34.399176 19569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96a74aab7e64898a040f093333f500b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:34.404141 19942 tablet_bootstrap.cc:492] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98: Bootstrap starting.
I20260926 14:50:34.405934 19944 tablet_bootstrap.cc:492] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247: Bootstrap starting.
I20260926 14:50:34.406153 19942 tablet_bootstrap.cc:654] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:34.406273 19943 tablet_bootstrap.cc:492] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476: Bootstrap starting.
I20260926 14:50:34.407050 19942 log.cc:824] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:34.407933 19944 tablet_bootstrap.cc:654] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:34.408010 19943 tablet_bootstrap.cc:654] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:34.408663 19943 log.cc:824] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:34.409164 19944 log.cc:824] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:34.409294 19942 tablet_bootstrap.cc:492] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98: No bootstrap required, opened a new log
I20260926 14:50:34.409468 19942 ts_tablet_manager.cc:1397] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:50:34.410248 19943 tablet_bootstrap.cc:492] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476: No bootstrap required, opened a new log
I20260926 14:50:34.410406 19943 ts_tablet_manager.cc:1397] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 14:50:34.410938 19944 tablet_bootstrap.cc:492] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247: No bootstrap required, opened a new log
I20260926 14:50:34.411101 19944 ts_tablet_manager.cc:1397] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:50:34.413544 19943 raft_consensus.cc:348] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } }
I20260926 14:50:34.413751 19943 raft_consensus.cc:374] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:34.413604 19942 raft_consensus.cc:348] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } }
I20260926 14:50:34.413800 19943 raft_consensus.cc:729] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687fc8c182854d2da4f7d2a7418ce476, State: Initialized, Role: FOLLOWER
I20260926 14:50:34.413820 19942 raft_consensus.cc:374] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:34.413864 19942 raft_consensus.cc:729] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b60ac486fb400e95bb50df4ed3eb98, State: Initialized, Role: FOLLOWER
I20260926 14:50:34.414193 19943 consensus_queue.cc:260] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } }
I20260926 14:50:34.414341 19942 consensus_queue.cc:260] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } }
I20260926 14:50:34.415186 19943 ts_tablet_manager.cc:1428] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260926 14:50:34.415206 19942 ts_tablet_manager.cc:1428] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260926 14:50:34.415128 19944 raft_consensus.cc:348] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } }
I20260926 14:50:34.415344 19944 raft_consensus.cc:374] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:34.415402 19944 raft_consensus.cc:729] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c41ef8146da478c86767ede40d05247, State: Initialized, Role: FOLLOWER
I20260926 14:50:34.415530 19773 heartbeater.cc:503] Master 127.12.68.254:43839 was elected leader, sending a full tablet report...
I20260926 14:50:34.415800 19944 consensus_queue.cc:260] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } }
I20260926 14:50:34.415855 19909 heartbeater.cc:503] Master 127.12.68.254:43839 was elected leader, sending a full tablet report...
I20260926 14:50:34.416692 19944 ts_tablet_manager.cc:1428] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260926 14:50:34.418069 19636 heartbeater.cc:503] Master 127.12.68.254:43839 was elected leader, sending a full tablet report...
W20260926 14:50:34.422660 19775 tablet.cc:2369] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:50:34.480769 19638 tablet.cc:2369] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:50:34.592133 19911 tablet.cc:2369] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:34.600011 19948 raft_consensus.cc:482] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:34.600194 19948 raft_consensus.cc:504] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } }
I20260926 14:50:34.601267 19948 leader_election.cc:288] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c41ef8146da478c86767ede40d05247 (127.12.68.193:38767), f7b60ac486fb400e95bb50df4ed3eb98 (127.12.68.194:45333)
I20260926 14:50:34.606029 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.601631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:57513 (local address 127.12.68.193:38767)
0926 14:50:34.601764 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.601768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.601816 (+ 48us) server_negotiation.cc:407] Connection header received
0926 14:50:34.601976 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.601979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.602025 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.602112 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:34.602424 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.603205 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.604809 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.605421 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.605541 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.605590 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.605835 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.605904 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:50:34.606029 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.601631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:54093 (local address 127.12.68.194:45333)
0926 14:50:34.601748 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.601753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.601816 (+ 63us) server_negotiation.cc:407] Connection header received
0926 14:50:34.601976 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.601979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.602025 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.602112 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:34.602424 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.603205 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.604809 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.605421 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.605541 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.605590 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.605835 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.605904 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:50:34.606051 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.601557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:38767 (local address 127.12.68.195:57513)
0926 14:50:34.601746 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.601761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.601862 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.602134 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.602137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.602160 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:34.602318 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.602325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.603314 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.603318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.604670 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.604681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.605543 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.605546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.605715 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.605749 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.605810 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.605898 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":184,"spinlock_wait_cycles":15104}
I20260926 14:50:34.606050 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.601557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:45333 (local address 127.12.68.195:54093)
0926 14:50:34.601746 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.601761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.601862 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.602134 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.602137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.602160 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:34.602321 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.602326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.603314 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.603318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.604673 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.604681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.605554 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.605557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.605689 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.605735 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.605810 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.605898 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":120,"spinlock_wait_cycles":5248}
I20260926 14:50:34.606709 19726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96a74aab7e64898a040f093333f500b" candidate_uuid: "687fc8c182854d2da4f7d2a7418ce476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" is_pre_election: true
I20260926 14:50:34.606709 19589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96a74aab7e64898a040f093333f500b" candidate_uuid: "687fc8c182854d2da4f7d2a7418ce476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c41ef8146da478c86767ede40d05247" is_pre_election: true
I20260926 14:50:34.607010 19726 raft_consensus.cc:2454] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 687fc8c182854d2da4f7d2a7418ce476 in term 0.
I20260926 14:50:34.607010 19589 raft_consensus.cc:2454] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 687fc8c182854d2da4f7d2a7418ce476 in term 0.
I20260926 14:50:34.607446 19797 leader_election.cc:302] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 687fc8c182854d2da4f7d2a7418ce476, 9c41ef8146da478c86767ede40d05247; no voters:
I20260926 14:50:34.607689 19948 raft_consensus.cc:2781] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:34.607774 19948 raft_consensus.cc:482] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:34.607822 19948 raft_consensus.cc:3037] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:34.608825 19948 raft_consensus.cc:504] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } }
I20260926 14:50:34.609128 19948 leader_election.cc:288] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [CANDIDATE]: Term 1 election: Requested vote from peers 9c41ef8146da478c86767ede40d05247 (127.12.68.193:38767), f7b60ac486fb400e95bb50df4ed3eb98 (127.12.68.194:45333)
I20260926 14:50:34.609419 19589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96a74aab7e64898a040f093333f500b" candidate_uuid: "687fc8c182854d2da4f7d2a7418ce476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c41ef8146da478c86767ede40d05247"
I20260926 14:50:34.609438 19726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96a74aab7e64898a040f093333f500b" candidate_uuid: "687fc8c182854d2da4f7d2a7418ce476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7b60ac486fb400e95bb50df4ed3eb98"
I20260926 14:50:34.609524 19589 raft_consensus.cc:3037] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:34.609563 19726 raft_consensus.cc:3037] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:34.610419 19589 raft_consensus.cc:2454] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687fc8c182854d2da4f7d2a7418ce476 in term 1.
I20260926 14:50:34.610520 19726 raft_consensus.cc:2454] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687fc8c182854d2da4f7d2a7418ce476 in term 1.
I20260926 14:50:34.610695 19797 leader_election.cc:302] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 687fc8c182854d2da4f7d2a7418ce476, 9c41ef8146da478c86767ede40d05247; no voters:
I20260926 14:50:34.610881 19948 raft_consensus.cc:2781] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:34.611076 19948 raft_consensus.cc:686] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [term 1 LEADER]: Becoming Leader. State: Replica: 687fc8c182854d2da4f7d2a7418ce476, State: Running, Role: LEADER
I20260926 14:50:34.611384 19948 consensus_queue.cc:237] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } }
I20260926 14:50:34.614672 18670 catalog_manager.cc:5495] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 reported cstate change: term changed from 0 to 1, leader changed from <none> to 687fc8c182854d2da4f7d2a7418ce476 (127.12.68.195). New cstate: current_term: 1 leader_uuid: "687fc8c182854d2da4f7d2a7418ce476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } health_report { overall_health: HEALTHY } } }
I20260926 14:50:34.680790 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.677390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.12.68.195:44785)
0926 14:50:34.677508 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.677512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.677527 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:50:34.677641 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.677643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.677695 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.677798 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:34.678024 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.678537 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.679372 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.679581 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.679665 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.679966 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:34.680374 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:34.680480 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.680553 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.680607 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:50:34.686264 19726 raft_consensus.cc:1261] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98 [term 1 FOLLOWER]: Refusing update from remote peer 687fc8c182854d2da4f7d2a7418ce476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:34.686267 19589 raft_consensus.cc:1261] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247 [term 1 FOLLOWER]: Refusing update from remote peer 687fc8c182854d2da4f7d2a7418ce476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:34.686916 19951 consensus_queue.cc:1035] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:34.687078 19948 consensus_queue.cc:1035] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:34.694495 19954 mvcc.cc:205] Tried to move back new op lower bound from 7333618625268662272 to 7333618624970596352. Current Snapshot: MvccSnapshot[applied={T|T < 7333618625268662272}]
I20260926 14:50:34.698975 19956 mvcc.cc:205] Tried to move back new op lower bound from 7333618625268662272 to 7333618624970596352. Current Snapshot: MvccSnapshot[applied={T|T < 7333618625268662272}]
I20260926 14:50:34.704762 18671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:34.704893 18671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:34.708829 18671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccf70cbde5ec4195a72786c4dd8b3e5a: Sending DeleteTablet for 3 replicas of tablet a96a74aab7e64898a040f093333f500b
I20260926 14:50:34.709489 19706 tablet_service.cc:1514] Processing DeleteTablet for tablet a96a74aab7e64898a040f093333f500b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:34 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:60174
I20260926 14:50:34.709506 19569 tablet_service.cc:1514] Processing DeleteTablet for tablet a96a74aab7e64898a040f093333f500b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:34 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:36630
I20260926 14:50:34.709888 19965 tablet_replica.cc:330] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98: stopping tablet replica
I20260926 14:50:34.709918 19842 tablet_service.cc:1514] Processing DeleteTablet for tablet a96a74aab7e64898a040f093333f500b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:34 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:38814
I20260926 14:50:34.710155 19965 raft_consensus.cc:2229] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:34.710347 19966 tablet_replica.cc:330] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476: stopping tablet replica
I20260926 14:50:34.710460 19965 raft_consensus.cc:2258] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:34.710686 19966 raft_consensus.cc:2229] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:34.711027 19966 raft_consensus.cc:2258] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:34.712007 19965 ts_tablet_manager.cc:1905] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:34.712284 19966 ts_tablet_manager.cc:1905] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:34.713742 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 18640
I20260926 14:50:34.714567 19965 ts_tablet_manager.cc:1918] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:34.714664 19965 log.cc:1196] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/a96a74aab7e64898a040f093333f500b
I20260926 14:50:34.714916 19966 ts_tablet_manager.cc:1918] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:34.714955 19965 ts_tablet_manager.cc:1939] T a96a74aab7e64898a040f093333f500b P f7b60ac486fb400e95bb50df4ed3eb98: Deleting consensus metadata
I20260926 14:50:34.715067 19966 log.cc:1196] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/a96a74aab7e64898a040f093333f500b
I20260926 14:50:34.715380 19966 ts_tablet_manager.cc:1939] T a96a74aab7e64898a040f093333f500b P 687fc8c182854d2da4f7d2a7418ce476: Deleting consensus metadata
W20260926 14:50:34.720656 19795 connection.cc:535] server connection from 127.0.0.1:38814 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 14:50:34.720911 19660 connection.cc:535] server connection from 127.0.0.1:60174 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:50:34.721949 19964 tablet_replica.cc:330] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247: stopping tablet replica
I20260926 14:50:34.722234 19964 raft_consensus.cc:2229] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:34.722510 19964 raft_consensus.cc:2258] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:34.723871 19964 ts_tablet_manager.cc:1905] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:34.724160 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.711308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.12.68.253:44171)
0926 14:50:34.711407 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.711411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.715081 (+ 3670us) server_negotiation.cc:407] Connection header received
0926 14:50:34.715248 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.715251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.715292 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.715348 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.716804 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.717490 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.718432 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.718709 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.718802 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.720471 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.720474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.720477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.720501 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.722103 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.722856 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.722859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.722861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.722946 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.723233 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.723248 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.723250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.723449 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.723561 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.723904 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.724015 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:50:34.726353 19964 ts_tablet_manager.cc:1918] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:34.726444 19964 log.cc:1196] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-0/wal/wals/a96a74aab7e64898a040f093333f500b
I20260926 14:50:34.726740 19964 ts_tablet_manager.cc:1939] T a96a74aab7e64898a040f093333f500b P 9c41ef8146da478c86767ede40d05247: Deleting consensus metadata
W20260926 14:50:34.727658 19524 connection.cc:441] server connection from 127.0.0.1:36630 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36630 (request call id 1) could send its response
I20260926 14:50:34.733963 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.723818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.12.68.253:44171)
0926 14:50:34.723940 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.723943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.724935 (+ 992us) server_negotiation.cc:407] Connection header received
0926 14:50:34.725085 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.725088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.725124 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.725165 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.726024 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.726731 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.727832 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.728165 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.728234 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.729874 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.729878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.729881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.729905 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.732195 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.732687 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.732691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.732693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.732773 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.733110 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.733113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.733114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.733308 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.733400 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.733715 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.733820 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:50:34.734869 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.711133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.12.68.252:46837)
0926 14:50:34.711238 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.711243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.725578 (+ 14335us) server_negotiation.cc:407] Connection header received
0926 14:50:34.725737 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.725740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.725807 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.725858 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.726766 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.727553 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.728921 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.729289 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.729370 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.730951 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.730955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.730957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.730982 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.732777 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.733212 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.733216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.733217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.733290 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.733571 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.733574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.733576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.733767 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.733873 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.734623 (+ 750us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.734730 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:50:34.736166 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.724406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35296 (local address 127.12.68.252:46837)
0926 14:50:34.724502 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.724506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.728081 (+ 3575us) server_negotiation.cc:407] Connection header received
0926 14:50:34.728228 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.728231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.728269 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.728312 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.729206 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.729991 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.731168 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.731470 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.731540 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.733008 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.733012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.733015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.733038 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.734543 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.735001 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.735004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.735006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.735091 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.735379 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.735382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.735384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.735575 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.735687 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.735923 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.736019 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:50:35.696964 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.696528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.194:43691)
0926 14:50:35.696794 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:35.696875 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":72,"threads_started":1}
W20260926 14:50:35.697280 19773 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:43839 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
I20260926 14:50:35.700554 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.700192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.193:38695)
0926 14:50:35.700391 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:35.700464 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":57,"threads_started":1}
W20260926 14:50:35.700911 19636 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:43839 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
I20260926 14:50:35.701220 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.700813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.195:50357)
0926 14:50:35.701060 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:35.701134 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":57,"threads_started":1}
W20260926 14:50:35.701427 19909 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:43839 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
I20260926 14:50:36.210083 19978 raft_consensus.cc:482] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ccf70cbde5ec4195a72786c4dd8b3e5a)
I20260926 14:50:36.210276 19978 raft_consensus.cc:504] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } }
I20260926 14:50:36.211243 19978 leader_election.cc:288] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ccf70cbde5ec4195a72786c4dd8b3e5a (127.12.68.254:43839), e7d0f6987f9e42e586b68f89d0bc95b4 (127.12.68.253:44171)
I20260926 14:50:36.211859 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.211391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.0.0.1:56004)
0926 14:50:36.211668 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.211751 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":27,"thread_start_us":58,"threads_started":1}
W20260926 14:50:36.212082 19442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
I20260926 14:50:36.212436 19983 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ccf70cbde5ec4195a72786c4dd8b3e5a)
I20260926 14:50:36.212606 19983 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } }
I20260926 14:50:36.212932 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.212647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.0.0.1:56018)
0926 14:50:36.212813 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.212868 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 14:50:36.213076 19442 leader_election.cc:334] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ccf70cbde5ec4195a72786c4dd8b3e5a (127.12.68.254:43839): Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
I20260926 14:50:36.213506 19983 leader_election.cc:288] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ccf70cbde5ec4195a72786c4dd8b3e5a (127.12.68.254:43839), de56ca7828f84d69902976cf729cc1cb (127.12.68.252:46837)
I20260926 14:50:36.214037 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.213587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.0.0.1:56028)
0926 14:50:36.213889 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.213956 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":50,"threads_started":1}
W20260926 14:50:36.214190 19258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
I20260926 14:50:36.214948 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.214744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.0.0.1:56032)
0926 14:50:36.214839 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.214888 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 14:50:36.215369 19258 leader_election.cc:334] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ccf70cbde5ec4195a72786c4dd8b3e5a (127.12.68.254:43839): Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
I20260926 14:50:36.215399 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.211391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44171 (local address 127.0.0.1:49144)
0926 14:50:36.211648 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.211664 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:36.211762 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:36.212064 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:36.212067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:36.212085 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:36.212177 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:36.212183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.213119 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.213122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:36.214209 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:36.214221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.215025 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.215028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:36.215113 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.215135 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:50:36.215207 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 14:50:36.215278 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":57,"threads_started":1}
I20260926 14:50:36.215399 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.211512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.12.68.253:44171)
0926 14:50:36.211800 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:36.211805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:36.211820 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:50:36.211871 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:36.211873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:36.211952 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:36.212046 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:36.212312 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.213030 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.214347 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.214924 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.215013 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.215069 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:36.215228 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 14:50:36.215296 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":64,"threads_started":1}
I20260926 14:50:36.216003 19281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de56ca7828f84d69902976cf729cc1cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" is_pre_election: true
I20260926 14:50:36.216182 19281 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de56ca7828f84d69902976cf729cc1cb in term 1.
I20260926 14:50:36.216493 19440 leader_election.cc:302] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: de56ca7828f84d69902976cf729cc1cb, e7d0f6987f9e42e586b68f89d0bc95b4; no voters: ccf70cbde5ec4195a72786c4dd8b3e5a
I20260926 14:50:36.216696 19978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 14:50:36.216755 19978 raft_consensus.cc:482] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 1 FOLLOWER]: Starting leader election (detected failure of leader ccf70cbde5ec4195a72786c4dd8b3e5a)
I20260926 14:50:36.216797 19978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:50:36.217491 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.213692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:46837 (local address 127.0.0.1:35300)
0926 14:50:36.213922 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.213935 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:36.214009 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:36.214298 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:36.214302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:36.214323 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:36.214407 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:36.214412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.215323 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.215326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:36.216403 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:36.216432 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.217163 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.217166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:36.217232 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.217251 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 14:50:36.217304 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 14:50:36.217367 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":59,"threads_started":1}
I20260926 14:50:36.217487 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.213804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.12.68.252:46837)
0926 14:50:36.214028 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:36.214036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:36.214067 (+ 31us) server_negotiation.cc:407] Connection header received
0926 14:50:36.214107 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:36.214109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:36.214149 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:36.214227 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:36.214525 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.215232 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.216515 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.217062 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.217169 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.217217 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:36.217327 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 14:50:36.217384 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":102,"threads_started":1}
I20260926 14:50:36.217973 19466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "de56ca7828f84d69902976cf729cc1cb" is_pre_election: true
I20260926 14:50:36.218092 19978 raft_consensus.cc:504] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } }
I20260926 14:50:36.218305 19466 raft_consensus.cc:2379] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e7d0f6987f9e42e586b68f89d0bc95b4 in current term 2: Already voted for candidate de56ca7828f84d69902976cf729cc1cb in this term.
I20260926 14:50:36.218394 19978 leader_election.cc:288] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [CANDIDATE]: Term 2 election: Requested vote from peers ccf70cbde5ec4195a72786c4dd8b3e5a (127.12.68.254:43839), e7d0f6987f9e42e586b68f89d0bc95b4 (127.12.68.253:44171)
I20260926 14:50:36.218708 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.218487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.0.0.1:56040)
0926 14:50:36.218611 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.218655 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 14:50:36.218720 19256 leader_election.cc:302] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7d0f6987f9e42e586b68f89d0bc95b4; no voters: ccf70cbde5ec4195a72786c4dd8b3e5a, de56ca7828f84d69902976cf729cc1cb
I20260926 14:50:36.218822 19281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de56ca7828f84d69902976cf729cc1cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4"
I20260926 14:50:36.218925 19281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:50:36.219319 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.219111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.0.0.1:56046)
0926 14:50:36.219209 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.219265 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:50:36.219416 19442 leader_election.cc:334] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ccf70cbde5ec4195a72786c4dd8b3e5a (127.12.68.254:43839): Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
I20260926 14:50:36.219916 19281 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de56ca7828f84d69902976cf729cc1cb in term 2.
I20260926 14:50:36.220026 19983 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:50:36.220170 19440 leader_election.cc:302] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: de56ca7828f84d69902976cf729cc1cb, e7d0f6987f9e42e586b68f89d0bc95b4; no voters: ccf70cbde5ec4195a72786c4dd8b3e5a
I20260926 14:50:36.220337 19978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:50:36.220505 19978 raft_consensus.cc:686] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [term 2 LEADER]: Becoming Leader. State: Replica: de56ca7828f84d69902976cf729cc1cb, State: Running, Role: LEADER
I20260926 14:50:36.220732 19978 consensus_queue.cc:237] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } }
I20260926 14:50:36.221848 19988 sys_catalog.cc:455] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader de56ca7828f84d69902976cf729cc1cb. Latest consensus state: current_term: 2 leader_uuid: "de56ca7828f84d69902976cf729cc1cb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } } }
I20260926 14:50:36.221951 19988 sys_catalog.cc:458] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:36.222186 19990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:36.224262 19990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:36.224624 19990 catalog_manager.cc:1259] Loaded cluster ID: 71d27facd073479e96c0cc5b21759065
I20260926 14:50:36.224668 19990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:36.225162 19990 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:36.225507 19990 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: Loaded TSK: 0
I20260926 14:50:36.226130 19990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:50:36.244736 19455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:36.247550 19455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:36.250365 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.250147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.0.0.1:56062)
0926 14:50:36.250247 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.250305 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 14:50:36.250435 19281 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [term 2 FOLLOWER]: Refusing update from remote peer de56ca7828f84d69902976cf729cc1cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 14:50:36.250895 19978 consensus_queue.cc:1035] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 14:50:36.251020 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.250826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.0.0.1:56072)
0926 14:50:36.250924 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.250968 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 14:50:36.251118 19442 consensus_peers.cc:487] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb -> Peer ccf70cbde5ec4195a72786c4dd8b3e5a (127.12.68.254:43839): Couldn't send request to peer ccf70cbde5ec4195a72786c4dd8b3e5a. Status: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 14:50:36.252364 19983 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de56ca7828f84d69902976cf729cc1cb. Latest consensus state: current_term: 2 leader_uuid: "de56ca7828f84d69902976cf729cc1cb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } } }
I20260926 14:50:36.252516 19983 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:50:36.253532 19988 sys_catalog.cc:455] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "de56ca7828f84d69902976cf729cc1cb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } } }
I20260926 14:50:36.253655 19988 sys_catalog.cc:458] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:36.253845 19983 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "de56ca7828f84d69902976cf729cc1cb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf70cbde5ec4195a72786c4dd8b3e5a" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43839 } } peers { permanent_uuid: "e7d0f6987f9e42e586b68f89d0bc95b4" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44171 } attrs { promote: false } } peers { permanent_uuid: "de56ca7828f84d69902976cf729cc1cb" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 46837 } attrs { promote: false } } }
I20260926 14:50:36.253957 19983 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d0f6987f9e42e586b68f89d0bc95b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:50:36.262995 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.259119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:44785 (local address 127.0.0.1:38834)
0926 14:50:36.259339 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.259351 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:36.259446 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:36.260197 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:36.260199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:36.260215 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:36.260290 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:36.260296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.261048 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.261051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:36.261878 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:36.261886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.262659 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.262663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:36.262751 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.262775 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:50:36.262818 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 14:50:36.262882 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260926 14:50:36.263396 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.258891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:38767 (local address 127.0.0.1:36636)
0926 14:50:36.258985 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.258998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:36.259071 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:36.259764 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:36.259767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:36.259783 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:36.259876 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:36.259903 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.260985 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.260988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:36.262141 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:36.262150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.263053 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.263056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:36.263143 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.263169 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:50:36.263223 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 14:50:36.263288 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":56}
I20260926 14:50:36.263512 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.259350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.12.68.195:44785)
0926 14:50:36.260000 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:36.260005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:36.260019 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:50:36.260065 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:36.260067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:36.260112 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:36.260190 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:36.260403 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.260952 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.261990 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.262536 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.263245 (+ 709us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.263303 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:36.263336 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 14:50:36.263402 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":64,"threads_started":1}
I20260926 14:50:36.263446 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.258986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.12.68.193:38767)
0926 14:50:36.259536 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:36.259540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:36.259553 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:36.259603 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:36.259606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:36.259654 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:36.259997 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:36.260109 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.260882 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.262363 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.262947 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.263045 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.263205 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:36.263251 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 14:50:36.263323 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":95,"threads_started":1}
I20260926 14:50:36.263908 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.259397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:45333 (local address 127.0.0.1:60176)
0926 14:50:36.259597 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.259609 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:36.259686 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:36.260342 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:36.260345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:36.260359 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:36.260431 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:36.260436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.261357 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.261360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:36.262455 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:36.262463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.263397 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.263401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:36.263632 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.263655 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:50:36.263716 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:50:36.263774 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":3,"thread_start_us":89,"threads_started":1}
I20260926 14:50:36.263962 19842 tablet_service.cc:1467] Processing CreateTablet for tablet 60427ed584da4516bd786ca9f1f4cd53 (DEFAULT_TABLE table=test-table [id=636147bd74cf42d1a7455769f85c8bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:36.264042 19569 tablet_service.cc:1467] Processing CreateTablet for tablet 60427ed584da4516bd786ca9f1f4cd53 (DEFAULT_TABLE table=test-table [id=636147bd74cf42d1a7455769f85c8bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:36.264165 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.259745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.12.68.194:45333)
0926 14:50:36.260083 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:36.260089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:36.260101 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:36.260208 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:36.260211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:36.260255 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:36.260337 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:36.260520 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.261251 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.262571 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.263217 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.263295 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.263350 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:36.263971 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 14:50:36.264044 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":63,"threads_started":1}
I20260926 14:50:36.264237 19842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60427ed584da4516bd786ca9f1f4cd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:36.264251 19569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60427ed584da4516bd786ca9f1f4cd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:36.266312 20001 tablet_bootstrap.cc:492] T 60427ed584da4516bd786ca9f1f4cd53 P 9c41ef8146da478c86767ede40d05247: Bootstrap starting.
I20260926 14:50:36.266698 19706 tablet_service.cc:1467] Processing CreateTablet for tablet 60427ed584da4516bd786ca9f1f4cd53 (DEFAULT_TABLE table=test-table [id=636147bd74cf42d1a7455769f85c8bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:36.266922 19706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60427ed584da4516bd786ca9f1f4cd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:36.267140 20001 tablet_bootstrap.cc:654] T 60427ed584da4516bd786ca9f1f4cd53 P 9c41ef8146da478c86767ede40d05247: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:36.267745 20002 tablet_bootstrap.cc:492] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476: Bootstrap starting.
I20260926 14:50:36.269060 20002 tablet_bootstrap.cc:654] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:36.269302 20004 tablet_bootstrap.cc:492] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98: Bootstrap starting.
I20260926 14:50:36.269531 20001 tablet_bootstrap.cc:492] T 60427ed584da4516bd786ca9f1f4cd53 P 9c41ef8146da478c86767ede40d05247: No bootstrap required, opened a new log
I20260926 14:50:36.269619 20001 ts_tablet_manager.cc:1397] T 60427ed584da4516bd786ca9f1f4cd53 P 9c41ef8146da478c86767ede40d05247: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:50:36.270092 20004 tablet_bootstrap.cc:654] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:36.270051 20001 raft_consensus.cc:348] T 60427ed584da4516bd786ca9f1f4cd53 P 9c41ef8146da478c86767ede40d05247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } }
I20260926 14:50:36.270180 20001 raft_consensus.cc:374] T 60427ed584da4516bd786ca9f1f4cd53 P 9c41ef8146da478c86767ede40d05247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:36.270216 20001 raft_consensus.cc:729] T 60427ed584da4516bd786ca9f1f4cd53 P 9c41ef8146da478c86767ede40d05247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c41ef8146da478c86767ede40d05247, State: Initialized, Role: FOLLOWER
I20260926 14:50:36.270330 20001 consensus_queue.cc:260] T 60427ed584da4516bd786ca9f1f4cd53 P 9c41ef8146da478c86767ede40d05247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } }
I20260926 14:50:36.270634 20001 ts_tablet_manager.cc:1428] T 60427ed584da4516bd786ca9f1f4cd53 P 9c41ef8146da478c86767ede40d05247: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:36.271276 20004 tablet_bootstrap.cc:492] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98: No bootstrap required, opened a new log
I20260926 14:50:36.271353 20004 ts_tablet_manager.cc:1397] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:36.271751 20004 raft_consensus.cc:348] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } }
I20260926 14:50:36.271845 20004 raft_consensus.cc:374] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:36.271875 20004 raft_consensus.cc:729] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b60ac486fb400e95bb50df4ed3eb98, State: Initialized, Role: FOLLOWER
I20260926 14:50:36.272032 20004 consensus_queue.cc:260] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } }
I20260926 14:50:36.272326 20004 ts_tablet_manager.cc:1428] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:36.272650 19635 heartbeater.cc:503] Master 127.12.68.252:46837 was elected leader, sending a full tablet report...
I20260926 14:50:36.273046 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.270861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.193:55655)
0926 14:50:36.272902 (+ 2041us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.272958 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1980,"thread_start_us":83,"threads_started":1}
I20260926 14:50:36.273658 19772 heartbeater.cc:503] Master 127.12.68.252:46837 was elected leader, sending a full tablet report...
I20260926 14:50:36.273969 20002 tablet_bootstrap.cc:492] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476: No bootstrap required, opened a new log
I20260926 14:50:36.274055 20002 ts_tablet_manager.cc:1397] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 14:50:36.274386 20002 raft_consensus.cc:348] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } }
I20260926 14:50:36.274492 20002 raft_consensus.cc:374] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:36.274544 20002 raft_consensus.cc:729] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687fc8c182854d2da4f7d2a7418ce476, State: Initialized, Role: FOLLOWER
I20260926 14:50:36.274654 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.273403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.193:44825)
0926 14:50:36.273513 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.274584 (+ 1071us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60,"thread_start_us":91,"threads_started":1}
I20260926 14:50:36.274667 20002 consensus_queue.cc:260] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } }
I20260926 14:50:36.275092 20002 ts_tablet_manager.cc:1428] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:36.275213 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.272937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.194:50307)
0926 14:50:36.275063 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.275117 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2064,"thread_start_us":81,"threads_started":1}
I20260926 14:50:36.275846 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.275407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.195:55559)
0926 14:50:36.275713 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.275765 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":94,"threads_started":1}
I20260926 14:50:36.276237 19908 heartbeater.cc:503] Master 127.12.68.252:46837 was elected leader, sending a full tablet report...
I20260926 14:50:36.276484 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.275792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.194:57731)
0926 14:50:36.276369 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.276416 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":117,"threads_started":1}
I20260926 14:50:36.276613 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.276204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.195:44425)
0926 14:50:36.276489 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.276548 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":66,"threads_started":1}
I20260926 14:50:36.542588 20009 raft_consensus.cc:482] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:36.542730 20009 raft_consensus.cc:504] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } }
I20260926 14:50:36.543931 20009 leader_election.cc:288] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c41ef8146da478c86767ede40d05247 (127.12.68.193:38767), 687fc8c182854d2da4f7d2a7418ce476 (127.12.68.195:44785)
I20260926 14:50:36.548254 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.544562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:48531 (local address 127.12.68.195:44785)
0926 14:50:36.544672 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:36.544676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:36.544687 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:50:36.544725 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:36.544727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:36.544760 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:36.544815 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:36.545031 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.545499 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.546996 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.547608 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.547714 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.547771 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:36.548053 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 14:50:36.548129 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:50:36.548254 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.544209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:42749 (local address 127.12.68.193:38767)
0926 14:50:36.544340 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:36.544345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:36.544359 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:50:36.544407 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:36.544410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:36.544449 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:36.544509 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:36.544770 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.545359 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.546996 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.547601 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.547714 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.547765 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:36.548053 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 14:50:36.548130 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:50:36.548285 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.544330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:44785 (local address 127.12.68.194:48531)
0926 14:50:36.544431 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.544444 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:36.544526 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:36.544841 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:36.544843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:36.544859 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:36.544937 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:36.544943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.545580 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.545583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:36.546875 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:36.546883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.547739 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.547743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:36.547870 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.547961 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 14:50:36.548037 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 14:50:36.548126 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":122}
I20260926 14:50:36.548291 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.544097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:38767 (local address 127.12.68.194:42749)
0926 14:50:36.544216 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.544229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:36.544328 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:36.544532 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:36.544535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:36.544549 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:36.544669 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:36.544674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.545455 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.545458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:36.546872 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:36.546883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.547739 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.547743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:36.547919 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.547966 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 14:50:36.548037 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 14:50:36.548128 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":148}
I20260926 14:50:36.548734 19589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60427ed584da4516bd786ca9f1f4cd53" candidate_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c41ef8146da478c86767ede40d05247" is_pre_election: true
I20260926 14:50:36.548862 19589 raft_consensus.cc:2454] T 60427ed584da4516bd786ca9f1f4cd53 P 9c41ef8146da478c86767ede40d05247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7b60ac486fb400e95bb50df4ed3eb98 in term 0.
I20260926 14:50:36.548926 19862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60427ed584da4516bd786ca9f1f4cd53" candidate_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687fc8c182854d2da4f7d2a7418ce476" is_pre_election: true
I20260926 14:50:36.549154 19862 raft_consensus.cc:2454] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7b60ac486fb400e95bb50df4ed3eb98 in term 0.
I20260926 14:50:36.549196 19661 leader_election.cc:302] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c41ef8146da478c86767ede40d05247, f7b60ac486fb400e95bb50df4ed3eb98; no voters:
I20260926 14:50:36.549444 20009 raft_consensus.cc:2781] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:36.549495 20009 raft_consensus.cc:482] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:36.549520 20009 raft_consensus.cc:3037] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:36.550482 20009 raft_consensus.cc:504] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } }
I20260926 14:50:36.550774 20009 leader_election.cc:288] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [CANDIDATE]: Term 1 election: Requested vote from peers 9c41ef8146da478c86767ede40d05247 (127.12.68.193:38767), 687fc8c182854d2da4f7d2a7418ce476 (127.12.68.195:44785)
I20260926 14:50:36.551158 19589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60427ed584da4516bd786ca9f1f4cd53" candidate_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c41ef8146da478c86767ede40d05247"
I20260926 14:50:36.551198 19862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60427ed584da4516bd786ca9f1f4cd53" candidate_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687fc8c182854d2da4f7d2a7418ce476"
I20260926 14:50:36.551263 19589 raft_consensus.cc:3037] T 60427ed584da4516bd786ca9f1f4cd53 P 9c41ef8146da478c86767ede40d05247 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:36.551283 19862 raft_consensus.cc:3037] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:36.552163 19589 raft_consensus.cc:2454] T 60427ed584da4516bd786ca9f1f4cd53 P 9c41ef8146da478c86767ede40d05247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7b60ac486fb400e95bb50df4ed3eb98 in term 1.
I20260926 14:50:36.552177 19862 raft_consensus.cc:2454] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7b60ac486fb400e95bb50df4ed3eb98 in term 1.
I20260926 14:50:36.552389 19661 leader_election.cc:302] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 687fc8c182854d2da4f7d2a7418ce476, f7b60ac486fb400e95bb50df4ed3eb98; no voters:
I20260926 14:50:36.552516 20009 raft_consensus.cc:2781] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:36.552726 20009 raft_consensus.cc:686] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [term 1 LEADER]: Becoming Leader. State: Replica: f7b60ac486fb400e95bb50df4ed3eb98, State: Running, Role: LEADER
I20260926 14:50:36.552940 20009 consensus_queue.cc:237] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } }
I20260926 14:50:36.553447 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.553216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.194:44311)
0926 14:50:36.553312 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.553363 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 14:50:36.555106 19455 catalog_manager.cc:5495] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7b60ac486fb400e95bb50df4ed3eb98 (127.12.68.194). New cstate: current_term: 1 leader_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7b60ac486fb400e95bb50df4ed3eb98" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45333 } health_report { overall_health: HEALTHY } } }
I20260926 14:50:36.620880 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.617788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.12.68.194:45333)
0926 14:50:36.617891 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:36.617896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:36.617911 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:50:36.618012 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:36.618014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:36.618065 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:36.618147 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:36.618362 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.618851 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.619613 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:36.619866 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:36.619953 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:36.620159 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:36.620589 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:36.620671 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:36.620734 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 14:50:36.620778 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 14:50:36.623618 19589 raft_consensus.cc:1261] T 60427ed584da4516bd786ca9f1f4cd53 P 9c41ef8146da478c86767ede40d05247 [term 1 FOLLOWER]: Refusing update from remote peer f7b60ac486fb400e95bb50df4ed3eb98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:36.623807 19862 raft_consensus.cc:1261] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476 [term 1 FOLLOWER]: Refusing update from remote peer f7b60ac486fb400e95bb50df4ed3eb98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:36.624120 20009 consensus_queue.cc:1035] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c41ef8146da478c86767ede40d05247" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 38767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:36.624342 20016 consensus_queue.cc:1035] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "687fc8c182854d2da4f7d2a7418ce476" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:36.627549 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.625837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.193:32873)
0926 14:50:36.627394 (+ 1557us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.627452 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1488}
I20260926 14:50:36.630393 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.630173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.194:45821)
0926 14:50:36.630277 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.630326 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 14:50:36.630731 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.629682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.193:37807)
0926 14:50:36.630625 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.630671 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":882}
I20260926 14:50:36.631052 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.626255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.195:39525)
0926 14:50:36.628167 (+ 1912us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.628223 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1597}
I20260926 14:50:36.631721 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.631480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.195:44081)
0926 14:50:36.631617 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.631665 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 14:50:36.634493 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.634273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43839 (local address 127.12.68.194:34687)
0926 14:50:36.634387 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.634433 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 14:50:36.636775 19456 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:36.636922 19456 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:36.640177 19456 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de56ca7828f84d69902976cf729cc1cb: Sending DeleteTablet for 3 replicas of tablet 60427ed584da4516bd786ca9f1f4cd53
I20260926 14:50:36.640913 19706 tablet_service.cc:1514] Processing DeleteTablet for tablet 60427ed584da4516bd786ca9f1f4cd53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:36 UTC) from {username='oryx', principal='oryx/127.12.68.252@KRBTEST.COM'} at 127.0.0.1:60176
I20260926 14:50:36.641074 19569 tablet_service.cc:1514] Processing DeleteTablet for tablet 60427ed584da4516bd786ca9f1f4cd53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:36 UTC) from {username='oryx', principal='oryx/127.12.68.252@KRBTEST.COM'} at 127.0.0.1:36636
I20260926 14:50:36.641400 20031 tablet_replica.cc:330] T 60427ed584da4516bd786ca9f1f4cd53 P 9c41ef8146da478c86767ede40d05247: stopping tablet replica
I20260926 14:50:36.641400 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 19416
I20260926 14:50:36.641471 19842 tablet_service.cc:1514] Processing DeleteTablet for tablet 60427ed584da4516bd786ca9f1f4cd53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:36 UTC) from {username='oryx', principal='oryx/127.12.68.252@KRBTEST.COM'} at 127.0.0.1:38834
I20260926 14:50:36.642606 20030 tablet_replica.cc:330] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98: stopping tablet replica
I20260926 14:50:36.642725 20030 raft_consensus.cc:2229] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:36.642987 20030 raft_consensus.cc:2258] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:36.643807 20030 ts_tablet_manager.cc:1905] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:36.646315 20030 ts_tablet_manager.cc:1918] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:36.646392 20030 log.cc:1196] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/60427ed584da4516bd786ca9f1f4cd53
I20260926 14:50:36.646644 20030 ts_tablet_manager.cc:1939] T 60427ed584da4516bd786ca9f1f4cd53 P f7b60ac486fb400e95bb50df4ed3eb98: Deleting consensus metadata
I20260926 14:50:36.647403 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 19643
I20260926 14:50:36.647571 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.647328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:38767 (local address 127.0.0.1:36642)
0926 14:50:36.647447 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.647501 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:38767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 14:50:36.648175 20032 tablet_replica.cc:330] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476: stopping tablet replica
I20260926 14:50:36.648285 20032 raft_consensus.cc:2229] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:36.648391 20032 raft_consensus.cc:2258] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:36.649108 20032 ts_tablet_manager.cc:1905] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 14:50:36.649209 19442 catalog_manager.cc:4584] TS 9c41ef8146da478c86767ede40d05247 (127.12.68.193:38767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60427ed584da4516bd786ca9f1f4cd53: Network error: Client connection negotiation failed: client connection to 127.12.68.193:38767: connect: Connection refused (error 111)
I20260926 14:50:36.650964 20032 ts_tablet_manager.cc:1918] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:36.651023 20032 log.cc:1196] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/60427ed584da4516bd786ca9f1f4cd53
I20260926 14:50:36.651237 20032 ts_tablet_manager.cc:1939] T 60427ed584da4516bd786ca9f1f4cd53 P 687fc8c182854d2da4f7d2a7418ce476: Deleting consensus metadata
I20260926 14:50:36.651760 19442 catalog_manager.cc:4841] TS 687fc8c182854d2da4f7d2a7418ce476 (127.12.68.195:44785): tablet 60427ed584da4516bd786ca9f1f4cd53 (table test-table [id=636147bd74cf42d1a7455769f85c8bf0]) successfully deleted
I20260926 14:50:36.652971 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 19779
I20260926 14:50:36.653898 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:36.653008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:45333 (local address 127.0.0.1:60196)
0926 14:50:36.653796 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:36.653840 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.194:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":728}
I20260926 14:50:36.658713 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 19136
I20260926 14:50:36.663102 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 19244
2026-09-26T14:50:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5390 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:36 dist-test-slave-24mn krb5kdc[20035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:36 dist-test-slave-24mn krb5kdc[20035](info): set up 2 sockets
Sep 26 14:50:36 dist-test-slave-24mn krb5kdc[20035](info): commencing operation
krb5kdc: starting...
W20260926 14:50:37.612161 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:50:37 dist-test-slave-24mn krb5kdc[20035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.254.keytab.
Entry for principal oryx/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.254.keytab.
I20260926 14:50:37.720525 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:33117
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.254.keytab
--principal=oryx/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:33117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:37.822656 20048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:37.822886 20048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:37.822938 20048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:37.826782 20048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:37.826830 20048 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 14:50:37.826853 20048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:37.826874 20048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:37.826896 20048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:37.826924 20048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:37.831974 20048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:33117
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.254.keytab
--principal=oryx/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:33117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:37.832878 20048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:37.833791 20048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:37.841691 20053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:37.841650 20056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:37.841660 20054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:37.841832 20048 server_base.cc:1034] running on GCE node
I20260926 14:50:37.842450 20048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:37.843441 20048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:37.844599 20048 hybrid_clock.cc:648] HybridClock initialized: now 1790434237844574 us; error 33 us; skew 500 ppm
Sep 26 14:50:37 dist-test-slave-24mn krb5kdc[20035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434237, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:37.847012 20048 init.cc:377] Logged in from keytab as oryx/127.12.68.254@KRBTEST.COM (short username oryx)
I20260926 14:50:37.847790 20048 webserver.cc:460] Webserver started at http://127.12.68.254:43709/ using document root <none> and password file <none>
I20260926 14:50:37.848306 20048 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:37.848363 20048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:37.848549 20048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:37.850432 20048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "58540ac4b0d4475eb9a19e933dc55553"
format_stamp: "Formatted at 2026-09-26 14:50:37 on dist-test-slave-24mn"
server_key: "109daf601c00be1275b1a6abb1ceae3b"
server_key_iv: "48e37605928119d9df4de07e5e177172"
server_key_version: "encryptionkey@0"
I20260926 14:50:37.850924 20048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "58540ac4b0d4475eb9a19e933dc55553"
format_stamp: "Formatted at 2026-09-26 14:50:37 on dist-test-slave-24mn"
server_key: "109daf601c00be1275b1a6abb1ceae3b"
server_key_iv: "48e37605928119d9df4de07e5e177172"
server_key_version: "encryptionkey@0"
I20260926 14:50:37.854403 20048 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:50:37.856680 20063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:37.857801 20048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 14:50:37.857908 20048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "58540ac4b0d4475eb9a19e933dc55553"
format_stamp: "Formatted at 2026-09-26 14:50:37 on dist-test-slave-24mn"
server_key: "109daf601c00be1275b1a6abb1ceae3b"
server_key_iv: "48e37605928119d9df4de07e5e177172"
server_key_version: "encryptionkey@0"
I20260926 14:50:37.857993 20048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:37.868780 20048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:37.871114 20048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:37.871264 20048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:37.878198 20048 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:33117
I20260926 14:50:37.878203 20125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:33117 every 8 connection(s)
I20260926 14:50:37.878997 20048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:37.881475 20126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:37.885275 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 20048
I20260926 14:50:37.885397 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:37.885666 12563 external_mini_cluster.cc:1428] Setting key 3ab7854a362a94385f9b8c819be48411
I20260926 14:50:37.886593 20126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553: Bootstrap starting.
I20260926 14:50:37.888638 20126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:37.889309 20126 log.cc:824] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:37.890954 20126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553: No bootstrap required, opened a new log
Sep 26 14:50:37 dist-test-slave-24mn krb5kdc[20035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
I20260926 14:50:37.894167 20126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58540ac4b0d4475eb9a19e933dc55553" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33117 } }
I20260926 14:50:37.894443 20126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:37.894500 20126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58540ac4b0d4475eb9a19e933dc55553, State: Initialized, Role: FOLLOWER
I20260926 14:50:37.894945 20126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58540ac4b0d4475eb9a19e933dc55553" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33117 } }
I20260926 14:50:37.895102 20126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:37.895167 20126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:37.895269 20126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:37.896342 20126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58540ac4b0d4475eb9a19e933dc55553" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33117 } }
I20260926 14:50:37.896663 20126 leader_election.cc:302] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58540ac4b0d4475eb9a19e933dc55553; no voters:
I20260926 14:50:37.896951 20126 leader_election.cc:288] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:37.897024 20131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:37.897231 20131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [term 1 LEADER]: Becoming Leader. State: Replica: 58540ac4b0d4475eb9a19e933dc55553, State: Running, Role: LEADER
I20260926 14:50:37.897504 20131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58540ac4b0d4475eb9a19e933dc55553" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33117 } }
I20260926 14:50:37.898140 20126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:37.898530 20133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58540ac4b0d4475eb9a19e933dc55553. Latest consensus state: current_term: 1 leader_uuid: "58540ac4b0d4475eb9a19e933dc55553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58540ac4b0d4475eb9a19e933dc55553" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33117 } } }
I20260926 14:50:37.898669 20133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:37.898885 20132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58540ac4b0d4475eb9a19e933dc55553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58540ac4b0d4475eb9a19e933dc55553" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33117 } } }
I20260926 14:50:37.898959 20132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:37.899062 20136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:37.899433 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:37.886847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49162 (local address 127.12.68.254:33117)
0926 14:50:37.887222 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:37.887240 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:37.887418 (+ 178us) server_negotiation.cc:407] Connection header received
0926 14:50:37.888136 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:37.888152 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:37.888453 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:37.888743 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:37.889467 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:37.890386 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:37.891214 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:37.891654 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:37.891793 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:37.893979 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:37.893984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:37.893996 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:37.894021 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:37.895948 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:37.896339 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:37.896345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:37.896350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:37.896485 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:37.896749 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:37.896752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:37.896754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:37.897217 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:37.898031 (+ 814us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:37.898173 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 14:50:37.898448 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":96,"threads_started":1}
I20260926 14:50:37.901985 20136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:37.907294 20136 catalog_manager.cc:1347] Generated new cluster ID: a28d9332cee0425290c4ea5ed512ca42
I20260926 14:50:37.907349 20136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:37.917122 20136 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:37.918640 20136 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:37.930691 20136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553: Generated new TSK 0
I20260926 14:50:37.931403 20136 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.193.keytab.
Entry for principal oryx/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.193.keytab.
I20260926 14:50:37.962311 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33117
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.193.keytab
--principal=oryx/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:38.065030 20152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:38.065263 20152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:38.065311 20152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:38.069237 20152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:38.069289 20152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:38.069335 20152 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:38.069389 20152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:38.074371 20152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.193.keytab
--principal=oryx/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.68.254:33117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:38.075304 20152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:38.076292 20152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:38.084997 20157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.084995 20160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.084995 20158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:38.085166 20152 server_base.cc:1034] running on GCE node
I20260926 14:50:38.085600 20152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:38.086153 20152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:38.087306 20152 hybrid_clock.cc:648] HybridClock initialized: now 1790434238087285 us; error 31 us; skew 500 ppm
Sep 26 14:50:38 dist-test-slave-24mn krb5kdc[20035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:38.089794 20152 init.cc:377] Logged in from keytab as oryx/127.12.68.193@KRBTEST.COM (short username oryx)
I20260926 14:50:38.090548 20152 webserver.cc:460] Webserver started at http://127.12.68.193:38923/ using document root <none> and password file <none>
I20260926 14:50:38.091032 20152 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:38.091090 20152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:38.091262 20152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:38.093125 20152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "39bfa3c4f11a47e6a4a22f8c51311cde"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-24mn"
server_key: "9e6a54be57bcceb0aa883a961a0ed2bb"
server_key_iv: "4cc00213a3f2bf6a140b89722a8dd434"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.093606 20152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "39bfa3c4f11a47e6a4a22f8c51311cde"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-24mn"
server_key: "9e6a54be57bcceb0aa883a961a0ed2bb"
server_key_iv: "4cc00213a3f2bf6a140b89722a8dd434"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.096912 20152 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:50:38.099273 20167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.100445 20152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:38.100536 20152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "39bfa3c4f11a47e6a4a22f8c51311cde"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-24mn"
server_key: "9e6a54be57bcceb0aa883a961a0ed2bb"
server_key_iv: "4cc00213a3f2bf6a140b89722a8dd434"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.100621 20152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:38.124768 20152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:38.127223 20152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:38.127372 20152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:38.128741 20152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:38.128786 20152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.128824 20152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:38.128849 20152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:50:38 dist-test-slave-24mn krb5kdc[20035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
I20260926 14:50:38.138471 20152 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:44621
I20260926 14:50:38.138523 20282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:44621 every 8 connection(s)
I20260926 14:50:38.139465 20152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:38.141139 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.129367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:32973 (local address 127.12.68.254:33117)
0926 14:50:38.129499 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.129504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.130342 (+ 838us) server_negotiation.cc:407] Connection header received
0926 14:50:38.131326 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.131331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.131388 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.131435 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.132779 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.133576 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.135015 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.135256 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.135343 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.137664 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:38.137668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:38.137670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:38.137695 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:38.139240 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.139716 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.139721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.139723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.139815 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.140229 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.140232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.140234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.140404 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:38.140474 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.140907 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.141012 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:50:38.141865 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.129753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:33117 (local address 127.12.68.193:32973)
0926 14:50:38.130178 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.130227 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.131088 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.131755 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.131764 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.132125 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.132635 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.132653 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.133746 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.133749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.134904 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.134914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.135400 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.135404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.135566 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.136086 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.136101 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.137503 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:38.139344 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.139348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.139359 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.139628 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.139939 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.139942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.139944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.140100 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.140491 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:38.140496 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:38.140701 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 14:50:38.140860 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 14:50:38.141086 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":92,"threads_started":1}
I20260926 14:50:38.143078 20283 heartbeater.cc:347] Connected to a master server at 127.12.68.254:33117
I20260926 14:50:38.143205 20283 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:38.143646 20283 heartbeater.cc:511] Master 127.12.68.254:33117 requested a full tablet report, sending...
I20260926 14:50:38.145195 20080 ts_manager.cc:194] Registered new tserver with Master: 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193:44621)
I20260926 14:50:38.146493 20080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:32973
I20260926 14:50:38.147330 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 20152
I20260926 14:50:38.147452 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:38.147761 12563 external_mini_cluster.cc:1428] Setting key b4407e947d96e49a80a210bc3024f891
I20260926 14:50:38.152235 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.144941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:33117 (local address 127.12.68.193:43167)
0926 14:50:38.145196 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.145219 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.145316 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.145527 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.145530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.145719 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.145830 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.145836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.146557 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.146561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.147332 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.147341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.147936 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.147939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.148010 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.148508 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.148520 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.149065 (+ 545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:38.150780 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.150783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.150785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.151019 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.151268 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.151271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.151273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.151360 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.151746 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:38.151749 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:38.151840 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 14:50:38.151958 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 14:50:38.152082 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20260926 14:50:38.152316 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.145032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:43167 (local address 127.12.68.254:33117)
0926 14:50:38.145193 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.145196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.145283 (+ 87us) server_negotiation.cc:407] Connection header received
0926 14:50:38.145390 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.145393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.145434 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.145495 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.145933 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.146441 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.147459 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.147726 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.147783 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.149206 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:38.149210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:38.149213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:38.149237 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:38.150665 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.151114 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.151117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.151118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.151191 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.151474 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.151477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.151478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.151649 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:38.151736 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.152013 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.152139 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":100}
WARNING: no policy specified for oryx/127.12.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.194.keytab.
Entry for principal oryx/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.194.keytab.
I20260926 14:50:38.171676 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33117
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.194.keytab
--principal=oryx/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:38.274880 20292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:38.275121 20292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:38.275164 20292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:38.279083 20292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:38.279145 20292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:38.279182 20292 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:38.279212 20292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:38.284257 20292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.194.keytab
--principal=oryx/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.68.254:33117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:38.285172 20292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:38.286123 20292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:50:38.295060 20292 server_base.cc:1034] running on GCE node
W20260926 14:50:38.294986 20297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.294986 20300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.294981 20298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:38.295559 20292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:38.296181 20292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:38.297322 20292 hybrid_clock.cc:648] HybridClock initialized: now 1790434238297296 us; error 38 us; skew 500 ppm
Sep 26 14:50:38 dist-test-slave-24mn krb5kdc[20035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:38.299993 20292 init.cc:377] Logged in from keytab as oryx/127.12.68.194@KRBTEST.COM (short username oryx)
I20260926 14:50:38.300789 20292 webserver.cc:460] Webserver started at http://127.12.68.194:35531/ using document root <none> and password file <none>
I20260926 14:50:38.301286 20292 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:38.301342 20292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:38.301517 20292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:38.303478 20292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "44d126fab62e44f998597b16f5ba6a65"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-24mn"
server_key: "063da02e7e331bd8e84e510ed19adaab"
server_key_iv: "81bdc59152eb54865495c3f823210889"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.304060 20292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "44d126fab62e44f998597b16f5ba6a65"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-24mn"
server_key: "063da02e7e331bd8e84e510ed19adaab"
server_key_iv: "81bdc59152eb54865495c3f823210889"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.307503 20292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:38.309731 20307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.310871 20292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:38.310969 20292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "44d126fab62e44f998597b16f5ba6a65"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-24mn"
server_key: "063da02e7e331bd8e84e510ed19adaab"
server_key_iv: "81bdc59152eb54865495c3f823210889"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.311058 20292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:38.324519 20292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:38.326931 20292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:38.327064 20292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:38.328332 20292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:38.328377 20292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.328415 20292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:38.328442 20292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:50:38 dist-test-slave-24mn krb5kdc[20035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
I20260926 14:50:38.337908 20292 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:33691
I20260926 14:50:38.337968 20422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:33691 every 8 connection(s)
I20260926 14:50:38.338749 20292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:38.341295 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.329147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:55023 (local address 127.12.68.254:33117)
0926 14:50:38.329263 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.329267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.330195 (+ 928us) server_negotiation.cc:407] Connection header received
0926 14:50:38.331165 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.331168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.331224 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.331293 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.332790 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.333595 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.335215 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.335528 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.335609 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.338276 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:38.338280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:38.338282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:38.338309 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:38.339736 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.340179 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.340182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.340183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.340256 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.340482 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.340484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.340485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.340658 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:38.340726 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.341091 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.341182 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:50:38.342006 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.329599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:33117 (local address 127.12.68.194:55023)
0926 14:50:38.330058 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.330096 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.330952 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.331512 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.331521 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.331993 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.332611 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.332640 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.333740 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.333744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.335109 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.335120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.335684 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.335688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.335875 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.336483 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.336502 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.338127 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:38.339819 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.339823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.339831 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.340096 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.340310 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.340312 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.340313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.340419 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.340727 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:38.340731 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:38.340920 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 14:50:38.341050 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 14:50:38.341271 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":128,"threads_started":1}
I20260926 14:50:38.343044 20423 heartbeater.cc:347] Connected to a master server at 127.12.68.254:33117
I20260926 14:50:38.343175 20423 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:38.343617 20423 heartbeater.cc:511] Master 127.12.68.254:33117 requested a full tablet report, sending...
I20260926 14:50:38.344558 20079 ts_manager.cc:194] Registered new tserver with Master: 44d126fab62e44f998597b16f5ba6a65 (127.12.68.194:33691)
I20260926 14:50:38.345224 20079 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:55023
I20260926 14:50:38.346957 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 20292
I20260926 14:50:38.347081 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:38.347411 12563 external_mini_cluster.cc:1428] Setting key 2c178a04541931f2c2647b24fbb0f081
I20260926 14:50:38.351990 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.345005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:59099 (local address 127.12.68.254:33117)
0926 14:50:38.345100 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.345104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.345618 (+ 514us) server_negotiation.cc:407] Connection header received
0926 14:50:38.345742 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.345745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.345787 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.345860 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.346204 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.346903 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.347928 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.348215 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.348309 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.349255 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:38.349259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:38.349261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:38.349283 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:38.350572 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.350964 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.350966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.350967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.351026 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.351254 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.351256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.351257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.351454 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:38.351523 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.351760 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.351840 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:50:38.352033 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.344942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:33117 (local address 127.12.68.194:59099)
0926 14:50:38.345545 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.345558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.345654 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.345872 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.345874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.346022 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.346110 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.346115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.347033 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.347036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.347811 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.347819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.348324 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.348326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.348396 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.348838 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.348850 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.349133 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:38.350666 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.350669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.350671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.350887 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.351089 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.351092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.351094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.351185 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.351535 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:38.351538 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:38.351618 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 14:50:38.351720 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:50:38.351838 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for oryx/127.12.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.195.keytab.
Entry for principal oryx/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.195.keytab.
I20260926 14:50:38.371750 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33117
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.195.keytab
--principal=oryx/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:38.476428 20432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:38.476666 20432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:38.476727 20432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:38.480636 20432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:38.480691 20432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:38.480743 20432 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:38.480798 20432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:38.485785 20432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.195.keytab
--principal=oryx/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.68.254:33117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:38.486727 20432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:38.487663 20432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:38.496341 20437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.496335 20438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:38.496399 20432 server_base.cc:1034] running on GCE node
W20260926 14:50:38.496341 20440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:38.496961 20432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:38.497540 20432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:38.498682 20432 hybrid_clock.cc:648] HybridClock initialized: now 1790434238498648 us; error 51 us; skew 500 ppm
Sep 26 14:50:38 dist-test-slave-24mn krb5kdc[20035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:38.501319 20432 init.cc:377] Logged in from keytab as oryx/127.12.68.195@KRBTEST.COM (short username oryx)
I20260926 14:50:38.502084 20432 webserver.cc:460] Webserver started at http://127.12.68.195:36167/ using document root <none> and password file <none>
I20260926 14:50:38.502574 20432 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:38.502638 20432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:38.502810 20432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:38.504784 20432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "8725b41178594da893bc86987ca15e1e"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-24mn"
server_key: "db88f4fe60e8bfd8d6801a59f038e117"
server_key_iv: "e28907b00c8c2a9b0c849fc04099716c"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.505380 20432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "8725b41178594da893bc86987ca15e1e"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-24mn"
server_key: "db88f4fe60e8bfd8d6801a59f038e117"
server_key_iv: "e28907b00c8c2a9b0c849fc04099716c"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.508904 20432 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:50:38.511140 20447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.512317 20432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:38.512419 20432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "8725b41178594da893bc86987ca15e1e"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-24mn"
server_key: "db88f4fe60e8bfd8d6801a59f038e117"
server_key_iv: "e28907b00c8c2a9b0c849fc04099716c"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.512504 20432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:38.523828 20432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:38.526546 20432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:38.526705 20432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:38.527938 20432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:38.527981 20432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.528028 20432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:38.528056 20432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:50:38 dist-test-slave-24mn krb5kdc[20035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
I20260926 14:50:38.537519 20432 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:46871
I20260926 14:50:38.537569 20562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:46871 every 8 connection(s)
I20260926 14:50:38.538486 20432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:38.540220 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.528677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:55243 (local address 127.12.68.254:33117)
0926 14:50:38.528781 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.528784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.529711 (+ 927us) server_negotiation.cc:407] Connection header received
0926 14:50:38.530675 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.530678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.530738 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.530796 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.532294 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.532868 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.533942 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.534216 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.534307 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.536756 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:38.536760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:38.536764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:38.536792 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:38.538313 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.538823 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.538827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.538829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.538921 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.539262 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.539264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.539265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.539439 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:38.539517 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.540002 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.540098 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:50:38.540944 20456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.529147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:33117 (local address 127.12.68.195:55243)
0926 14:50:38.529580 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.529618 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.530420 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.531173 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.531182 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.531573 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.532163 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.532181 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.532983 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.532987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.533830 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.533839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.534357 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.534361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.534529 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.535013 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.535028 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.536561 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:38.538428 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.538432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.538442 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.538724 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.539030 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.539032 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.539034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.539180 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.539546 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:38.539552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:38.539780 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 14:50:38.539964 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 14:50:38.540199 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260926 14:50:38.541971 20563 heartbeater.cc:347] Connected to a master server at 127.12.68.254:33117
I20260926 14:50:38.542088 20563 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:38.542516 20563 heartbeater.cc:511] Master 127.12.68.254:33117 requested a full tablet report, sending...
I20260926 14:50:38.543438 20080 ts_manager.cc:194] Registered new tserver with Master: 8725b41178594da893bc86987ca15e1e (127.12.68.195:46871)
I20260926 14:50:38.544128 20080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:55243
I20260926 14:50:38.547238 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 20432
I20260926 14:50:38.547358 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:38.547670 12563 external_mini_cluster.cc:1428] Setting key f1a2ded44ac295f2fcaa3073da12cb3d
I20260926 14:50:38.549629 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:50:38.550437 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.543956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:39995 (local address 127.12.68.254:33117)
0926 14:50:38.544043 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.544046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.544233 (+ 187us) server_negotiation.cc:407] Connection header received
0926 14:50:38.544372 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.544375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.544419 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.544498 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.544905 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.545373 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.546312 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.546523 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.546632 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.547539 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:38.547542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:38.547544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:38.547564 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:38.548907 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.549361 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.549365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.549367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.549458 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.549722 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.549725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.549727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.549887 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:38.549967 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.550212 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.550305 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 14:50:38.550447 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.543818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:33117 (local address 127.12.68.195:39995)
0926 14:50:38.544158 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.544172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.544290 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.544512 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.544515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.544673 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.544804 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.544811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.545489 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.545493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.546197 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.546205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.546632 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.546634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.546693 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.547059 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.547073 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.547427 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:38.548995 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.548998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.549000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.549236 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.549543 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.549546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.549548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.549637 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.549969 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:38.549972 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:38.550053 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 14:50:38.550141 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 14:50:38.550262 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":7,"thread_start_us":106,"threads_started":1}
Sep 26 14:50:38 dist-test-slave-24mn krb5kdc[20035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:50:38 dist-test-slave-24mn krb5kdc[20035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:38 dist-test-slave-24mn krb5kdc[20035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:50:38.571102 20129 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:38.565854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49174 (local address 127.12.68.254:33117)
0926 14:50:38.565949 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.565953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.565963 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:50:38.566000 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.566002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.566035 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.566098 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.566691 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.567177 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.567968 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.568171 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.568239 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.570990 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49174: BlockingRecv error: recv got EOF from 127.0.0.1:49174 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 14:50:38 dist-test-slave-24mn krb5kdc[20035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
I20260926 14:50:38.579070 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.572363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.12.68.254:33117)
0926 14:50:38.572474 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.572478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.572569 (+ 91us) server_negotiation.cc:407] Connection header received
0926 14:50:38.572694 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.572697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.572730 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.572785 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.573322 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.573784 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.574491 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.574689 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.574767 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.576681 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:38.576685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:38.576686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:38.576706 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:38.577750 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.578096 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.578099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.578101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.578166 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.578383 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.578386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.578388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.578535 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:38.578643 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.578840 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.578933 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:50:38.581501 20080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:38.584350 20080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:38.597688 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.592745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.12.68.194:33691)
0926 14:50:38.593117 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.593128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.593151 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:50:38.593216 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.593221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.593380 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.593503 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:38.593835 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.594878 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.596142 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.596943 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.597038 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.597188 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.597461 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.597547 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":92,"threads_started":1}
I20260926 14:50:38.597687 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.591772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:46871 (local address 127.0.0.1:45588)
0926 14:50:38.592199 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.592246 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.592362 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.593350 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.593356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.593378 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:38.593487 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.593492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.594831 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.594834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.596122 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.596133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.597057 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.597060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.597239 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.597316 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 14:50:38.597422 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 14:50:38.597547 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":160,"thread_start_us":66,"threads_started":1}
I20260926 14:50:38.597832 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.592253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.12.68.195:46871)
0926 14:50:38.592820 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.592830 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.592854 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:50:38.592924 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.592929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.593200 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.593348 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:38.593615 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.594727 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.596216 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.596938 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.597038 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.597188 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.597620 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.597707 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":103,"threads_started":1}
I20260926 14:50:38.598408 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.591849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44621 (local address 127.0.0.1:33458)
0926 14:50:38.592948 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.592961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.593044 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.593660 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.593663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.593679 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:38.593757 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.593762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.594648 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.594651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.596122 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.596133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.598062 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.598065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.598150 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.598171 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:50:38.598234 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:50:38.598294 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":60,"thread_start_us":97,"threads_started":1}
I20260926 14:50:38.598438 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.592460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:33691 (local address 127.0.0.1:37850)
0926 14:50:38.593039 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.593050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.593114 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.593615 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.593618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.593634 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:38.593717 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.593723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.594970 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.594973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.596032 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.596041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.597057 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.597061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.597237 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.597318 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 14:50:38.597422 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 14:50:38.597549 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":127,"thread_start_us":85,"threads_started":1}
I20260926 14:50:38.598680 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.592620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.12.68.193:44621)
0926 14:50:38.593273 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.593282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.593303 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:50:38.593372 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.593376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.593556 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.593683 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:38.593852 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.594545 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.597244 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.597942 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.598036 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.598167 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.598258 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.598383 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":97,"threads_started":1}
I20260926 14:50:38.600699 20497 tablet_service.cc:1467] Processing CreateTablet for tablet 8ff84e3e9da34fe6ac8f2b10ce7383e5 (DEFAULT_TABLE table=test-table [id=8e617e163c604fc9a480a7b5fcc59ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:38.601550 20497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ff84e3e9da34fe6ac8f2b10ce7383e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:38.601214 20357 tablet_service.cc:1467] Processing CreateTablet for tablet 8ff84e3e9da34fe6ac8f2b10ce7383e5 (DEFAULT_TABLE table=test-table [id=8e617e163c604fc9a480a7b5fcc59ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:38.601214 20216 tablet_service.cc:1467] Processing CreateTablet for tablet 8ff84e3e9da34fe6ac8f2b10ce7383e5 (DEFAULT_TABLE table=test-table [id=8e617e163c604fc9a480a7b5fcc59ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:38.602169 20357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ff84e3e9da34fe6ac8f2b10ce7383e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:38.602169 20216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ff84e3e9da34fe6ac8f2b10ce7383e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:38.608180 20591 tablet_bootstrap.cc:492] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e: Bootstrap starting.
I20260926 14:50:38.610244 20591 tablet_bootstrap.cc:654] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:38.610908 20591 log.cc:824] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:38.611438 20592 tablet_bootstrap.cc:492] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde: Bootstrap starting.
I20260926 14:50:38.611950 20593 tablet_bootstrap.cc:492] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65: Bootstrap starting.
I20260926 14:50:38.612829 20591 tablet_bootstrap.cc:492] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e: No bootstrap required, opened a new log
I20260926 14:50:38.613008 20591 ts_tablet_manager.cc:1397] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:50:38.613947 20593 tablet_bootstrap.cc:654] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:38.614706 20593 log.cc:824] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:38.616348 20593 tablet_bootstrap.cc:492] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65: No bootstrap required, opened a new log
I20260926 14:50:38.616508 20593 ts_tablet_manager.cc:1397] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:50:38.616699 20591 raft_consensus.cc:348] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } }
I20260926 14:50:38.616905 20592 tablet_bootstrap.cc:654] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:38.616953 20591 raft_consensus.cc:374] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:38.617004 20591 raft_consensus.cc:729] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8725b41178594da893bc86987ca15e1e, State: Initialized, Role: FOLLOWER
I20260926 14:50:38.617445 20591 consensus_queue.cc:260] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } }
I20260926 14:50:38.617621 20592 log.cc:824] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:38.618086 20563 heartbeater.cc:503] Master 127.12.68.254:33117 was elected leader, sending a full tablet report...
I20260926 14:50:38.618433 20591 ts_tablet_manager.cc:1428] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:50:38.619765 20593 raft_consensus.cc:348] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } }
I20260926 14:50:38.620039 20593 raft_consensus.cc:374] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:38.620106 20593 raft_consensus.cc:729] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d126fab62e44f998597b16f5ba6a65, State: Initialized, Role: FOLLOWER
I20260926 14:50:38.620553 20592 tablet_bootstrap.cc:492] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde: No bootstrap required, opened a new log
I20260926 14:50:38.620553 20593 consensus_queue.cc:260] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } }
I20260926 14:50:38.620728 20592 ts_tablet_manager.cc:1397] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20260926 14:50:38.621129 20423 heartbeater.cc:503] Master 127.12.68.254:33117 was elected leader, sending a full tablet report...
I20260926 14:50:38.621479 20593 ts_tablet_manager.cc:1428] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:50:38.624068 20592 raft_consensus.cc:348] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } }
I20260926 14:50:38.624307 20592 raft_consensus.cc:374] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:38.624366 20592 raft_consensus.cc:729] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39bfa3c4f11a47e6a4a22f8c51311cde, State: Initialized, Role: FOLLOWER
I20260926 14:50:38.624727 20592 consensus_queue.cc:260] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } }
I20260926 14:50:38.625264 20283 heartbeater.cc:503] Master 127.12.68.254:33117 was elected leader, sending a full tablet report...
I20260926 14:50:38.625517 20592 ts_tablet_manager.cc:1428] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260926 14:50:38.640547 20284 tablet.cc:2369] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:38.747978 20598 raft_consensus.cc:482] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:38.748167 20598 raft_consensus.cc:504] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } }
I20260926 14:50:38.749249 20598 leader_election.cc:288] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8725b41178594da893bc86987ca15e1e (127.12.68.195:46871), 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193:44621)
I20260926 14:50:38.752892 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.749470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:46871 (local address 127.12.68.194:55841)
0926 14:50:38.749667 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.749682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.749773 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.750065 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.750068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.750088 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:38.750205 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.750212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.750971 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.750975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.752001 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.752012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.752584 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.752587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.752670 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.752688 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:50:38.752741 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 14:50:38.752793 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 14:50:38.752948 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.749644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:55841 (local address 127.12.68.195:46871)
0926 14:50:38.749752 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.749756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.749769 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:38.749911 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.749914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.749962 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.750051 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:38.750308 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.750868 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.752112 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.752483 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.752578 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.752630 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.752764 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.752828 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:50:38.753580 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.749694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:33243 (local address 127.12.68.193:44621)
0926 14:50:38.749812 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.749816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.750244 (+ 428us) server_negotiation.cc:407] Connection header received
0926 14:50:38.750333 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.750335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.750380 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.750451 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:38.751008 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.751459 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.752606 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.753092 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.753200 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.753243 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.753411 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.753495 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:50:38.753580 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.749470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44621 (local address 127.12.68.194:33243)
0926 14:50:38.750151 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.750164 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.750252 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.750464 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.750467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.750484 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:38.750898 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.750904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.751563 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.751566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.752487 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.752494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.753201 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.753205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.753291 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.753311 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:50:38.753384 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:50:38.753446 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":356,"thread_start_us":107,"threads_started":1}
I20260926 14:50:38.753595 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ff84e3e9da34fe6ac8f2b10ce7383e5" candidate_uuid: "44d126fab62e44f998597b16f5ba6a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8725b41178594da893bc86987ca15e1e" is_pre_election: true
I20260926 14:50:38.753870 20517 raft_consensus.cc:2454] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d126fab62e44f998597b16f5ba6a65 in term 0.
I20260926 14:50:38.754226 20237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ff84e3e9da34fe6ac8f2b10ce7383e5" candidate_uuid: "44d126fab62e44f998597b16f5ba6a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" is_pre_election: true
I20260926 14:50:38.754308 20308 leader_election.cc:302] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d126fab62e44f998597b16f5ba6a65, 8725b41178594da893bc86987ca15e1e; no voters:
I20260926 14:50:38.754480 20237 raft_consensus.cc:2454] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d126fab62e44f998597b16f5ba6a65 in term 0.
I20260926 14:50:38.754565 20598 raft_consensus.cc:2781] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:38.754650 20598 raft_consensus.cc:482] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:38.754698 20598 raft_consensus.cc:3037] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:38.755661 20598 raft_consensus.cc:504] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } }
I20260926 14:50:38.755980 20598 leader_election.cc:288] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [CANDIDATE]: Term 1 election: Requested vote from peers 8725b41178594da893bc86987ca15e1e (127.12.68.195:46871), 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193:44621)
I20260926 14:50:38.756269 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ff84e3e9da34fe6ac8f2b10ce7383e5" candidate_uuid: "44d126fab62e44f998597b16f5ba6a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8725b41178594da893bc86987ca15e1e"
I20260926 14:50:38.756285 20237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ff84e3e9da34fe6ac8f2b10ce7383e5" candidate_uuid: "44d126fab62e44f998597b16f5ba6a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde"
I20260926 14:50:38.756372 20517 raft_consensus.cc:3037] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:38.756381 20237 raft_consensus.cc:3037] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:38.757296 20237 raft_consensus.cc:2454] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d126fab62e44f998597b16f5ba6a65 in term 1.
I20260926 14:50:38.757539 20517 raft_consensus.cc:2454] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d126fab62e44f998597b16f5ba6a65 in term 1.
I20260926 14:50:38.757562 20310 leader_election.cc:302] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39bfa3c4f11a47e6a4a22f8c51311cde, 44d126fab62e44f998597b16f5ba6a65; no voters:
I20260926 14:50:38.757690 20598 raft_consensus.cc:2781] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:38.757902 20598 raft_consensus.cc:686] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [term 1 LEADER]: Becoming Leader. State: Replica: 44d126fab62e44f998597b16f5ba6a65, State: Running, Role: LEADER
I20260926 14:50:38.758194 20598 consensus_queue.cc:237] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } }
I20260926 14:50:38.761186 20080 catalog_manager.cc:5495] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d126fab62e44f998597b16f5ba6a65 (127.12.68.194). New cstate: current_term: 1 leader_uuid: "44d126fab62e44f998597b16f5ba6a65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } health_report { overall_health: HEALTHY } } }
W20260926 14:50:38.789355 20564 tablet.cc:2369] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:38.794139 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.790689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.12.68.194:33691)
0926 14:50:38.790800 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.790804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.790816 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:38.790907 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.790909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.790958 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.791039 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:38.791211 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.791712 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.792538 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.792910 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.793015 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.793250 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:38.793800 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:38.793909 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.793947 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.794005 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:50:38.800369 20517 raft_consensus.cc:1261] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Refusing update from remote peer 44d126fab62e44f998597b16f5ba6a65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:38.800370 20237 raft_consensus.cc:1261] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Refusing update from remote peer 44d126fab62e44f998597b16f5ba6a65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:38.800921 20601 consensus_queue.cc:1035] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:38.801105 20598 consensus_queue.cc:1035] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:38.808085 20604 mvcc.cc:205] Tried to move back new op lower bound from 7333618642119589888 to 7333618641955766272. Current Snapshot: MvccSnapshot[applied={T|T < 7333618642119589888}]
I20260926 14:50:38.808502 20606 mvcc.cc:205] Tried to move back new op lower bound from 7333618642119589888 to 7333618641955766272. Current Snapshot: MvccSnapshot[applied={T|T < 7333618642119589888}]
I20260926 14:50:38.809063 20605 mvcc.cc:205] Tried to move back new op lower bound from 7333618642119589888 to 7333618641955766272. Current Snapshot: MvccSnapshot[applied={T|T < 7333618642119589888}]
I20260926 14:50:38.815425 20080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:38.815550 20080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:38.817776 20080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553: Sending DeleteTablet for 3 replicas of tablet 8ff84e3e9da34fe6ac8f2b10ce7383e5
I20260926 14:50:38.818449 20357 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ff84e3e9da34fe6ac8f2b10ce7383e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:38 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:37850
I20260926 14:50:38.818442 20216 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ff84e3e9da34fe6ac8f2b10ce7383e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:38 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:33458
I20260926 14:50:38.818838 20497 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ff84e3e9da34fe6ac8f2b10ce7383e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:38 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:45588
I20260926 14:50:38.819198 20617 tablet_replica.cc:330] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e: stopping tablet replica
I20260926 14:50:38.819310 20615 tablet_replica.cc:330] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde: stopping tablet replica
I20260926 14:50:38.819514 20617 raft_consensus.cc:2229] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:38.819531 20615 raft_consensus.cc:2229] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:38.819787 20615 raft_consensus.cc:2258] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:38.819788 20617 raft_consensus.cc:2258] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:38.819908 20080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:38.820292 20080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:38.820303 20616 tablet_replica.cc:330] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65: stopping tablet replica
I20260926 14:50:38.820629 20616 raft_consensus.cc:2229] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:38.821215 20617 ts_tablet_manager.cc:1905] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:38.821843 20615 ts_tablet_manager.cc:1905] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:38.823699 20617 ts_tablet_manager.cc:1918] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:38.823841 20617 log.cc:1196] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/8ff84e3e9da34fe6ac8f2b10ce7383e5
I20260926 14:50:38.824214 20617 ts_tablet_manager.cc:1939] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 8725b41178594da893bc86987ca15e1e: Deleting consensus metadata
I20260926 14:50:38.824242 20615 ts_tablet_manager.cc:1918] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:38.824332 20615 log.cc:1196] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/wal/wals/8ff84e3e9da34fe6ac8f2b10ce7383e5
I20260926 14:50:38.824618 20615 ts_tablet_manager.cc:1939] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 39bfa3c4f11a47e6a4a22f8c51311cde: Deleting consensus metadata
I20260926 14:50:38.825496 20064 catalog_manager.cc:4841] TS 8725b41178594da893bc86987ca15e1e (127.12.68.195:46871): tablet 8ff84e3e9da34fe6ac8f2b10ce7383e5 (table test-table [id=8e617e163c604fc9a480a7b5fcc59ec1]) successfully deleted
I20260926 14:50:38.825886 20066 catalog_manager.cc:4841] TS 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193:44621): tablet 8ff84e3e9da34fe6ac8f2b10ce7383e5 (table test-table [id=8e617e163c604fc9a480a7b5fcc59ec1]) successfully deleted
I20260926 14:50:38.825928 20497 tablet_service.cc:1467] Processing CreateTablet for tablet 7f810f3ddd5542e9af95bb51ad2a90d8 (DEFAULT_TABLE table=test-table [id=1a847960d12d4b8fa2b422f3d8a9ec05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:38.826150 20497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f810f3ddd5542e9af95bb51ad2a90d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:38.826133 20216 tablet_service.cc:1467] Processing CreateTablet for tablet 7f810f3ddd5542e9af95bb51ad2a90d8 (DEFAULT_TABLE table=test-table [id=1a847960d12d4b8fa2b422f3d8a9ec05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:38.826316 20216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f810f3ddd5542e9af95bb51ad2a90d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:38.828114 20591 tablet_bootstrap.cc:492] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e: Bootstrap starting.
I20260926 14:50:38.828158 20592 tablet_bootstrap.cc:492] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde: Bootstrap starting.
I20260926 14:50:38.825712 20357 tablet_service.cc:1467] Processing CreateTablet for tablet 7f810f3ddd5542e9af95bb51ad2a90d8 (DEFAULT_TABLE table=test-table [id=1a847960d12d4b8fa2b422f3d8a9ec05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:38.828682 20357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f810f3ddd5542e9af95bb51ad2a90d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:38.828954 20591 tablet_bootstrap.cc:654] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:38.829021 20592 tablet_bootstrap.cc:654] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:38.829891 20591 tablet_bootstrap.cc:492] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e: No bootstrap required, opened a new log
I20260926 14:50:38.829959 20591 ts_tablet_manager.cc:1397] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:38.830379 20591 raft_consensus.cc:348] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:38.830472 20591 raft_consensus.cc:374] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:38.830500 20591 raft_consensus.cc:729] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8725b41178594da893bc86987ca15e1e, State: Initialized, Role: FOLLOWER
I20260926 14:50:38.830607 20591 consensus_queue.cc:260] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:38.830822 20591 ts_tablet_manager.cc:1428] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:38.830955 20593 tablet_bootstrap.cc:492] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65: Bootstrap starting.
I20260926 14:50:38.831753 20593 tablet_bootstrap.cc:654] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:38.825712 20616 raft_consensus.cc:2258] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:38.833078 20593 tablet_bootstrap.cc:492] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65: No bootstrap required, opened a new log
I20260926 14:50:38.833174 20593 ts_tablet_manager.cc:1397] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:38.833434 20616 ts_tablet_manager.cc:1905] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:38.833577 20593 raft_consensus.cc:348] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:38.833666 20593 raft_consensus.cc:374] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:38.833686 20593 raft_consensus.cc:729] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d126fab62e44f998597b16f5ba6a65, State: Initialized, Role: FOLLOWER
I20260926 14:50:38.833781 20593 consensus_queue.cc:260] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:38.833866 20601 raft_consensus.cc:482] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:38.833921 20601 raft_consensus.cc:504] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:38.834007 20593 ts_tablet_manager.cc:1428] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:50:38.834254 20601 leader_election.cc:288] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193:44621), 8725b41178594da893bc86987ca15e1e (127.12.68.195:46871)
I20260926 14:50:38.834497 20237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f810f3ddd5542e9af95bb51ad2a90d8" candidate_uuid: "44d126fab62e44f998597b16f5ba6a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" is_pre_election: true
I20260926 14:50:38.834528 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f810f3ddd5542e9af95bb51ad2a90d8" candidate_uuid: "44d126fab62e44f998597b16f5ba6a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8725b41178594da893bc86987ca15e1e" is_pre_election: true
I20260926 14:50:38.834630 20517 raft_consensus.cc:2454] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d126fab62e44f998597b16f5ba6a65 in term 0.
I20260926 14:50:38.834859 20308 leader_election.cc:302] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d126fab62e44f998597b16f5ba6a65, 8725b41178594da893bc86987ca15e1e; no voters:
W20260926 14:50:38.834976 20310 leader_election.cc:341] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193:44621): Illegal state: must be running to vote when last-logged opid is not known
I20260926 14:50:38.835050 20601 raft_consensus.cc:2781] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:38.835103 20601 raft_consensus.cc:482] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:38.835137 20601 raft_consensus.cc:3037] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:38.836097 20601 raft_consensus.cc:504] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:38.836395 20601 leader_election.cc:288] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [CANDIDATE]: Term 1 election: Requested vote from peers 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193:44621), 8725b41178594da893bc86987ca15e1e (127.12.68.195:46871)
I20260926 14:50:38.836792 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f810f3ddd5542e9af95bb51ad2a90d8" candidate_uuid: "44d126fab62e44f998597b16f5ba6a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8725b41178594da893bc86987ca15e1e"
I20260926 14:50:38.836887 20517 raft_consensus.cc:3037] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:38.837425 20616 ts_tablet_manager.cc:1918] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:38.837483 20616 log.cc:1196] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/8ff84e3e9da34fe6ac8f2b10ce7383e5
I20260926 14:50:38.837668 20616 ts_tablet_manager.cc:1939] T 8ff84e3e9da34fe6ac8f2b10ce7383e5 P 44d126fab62e44f998597b16f5ba6a65: Deleting consensus metadata
I20260926 14:50:38.837767 20517 raft_consensus.cc:2454] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d126fab62e44f998597b16f5ba6a65 in term 1.
I20260926 14:50:38.838109 20308 leader_election.cc:302] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d126fab62e44f998597b16f5ba6a65, 8725b41178594da893bc86987ca15e1e; no voters:
I20260926 14:50:38.838295 20592 tablet_bootstrap.cc:492] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde: No bootstrap required, opened a new log
I20260926 14:50:38.838375 20592 ts_tablet_manager.cc:1397] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260926 14:50:38.838683 20237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f810f3ddd5542e9af95bb51ad2a90d8" candidate_uuid: "44d126fab62e44f998597b16f5ba6a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde"
I20260926 14:50:38.838768 20592 raft_consensus.cc:348] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:38.838851 20592 raft_consensus.cc:374] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:38.838871 20592 raft_consensus.cc:729] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39bfa3c4f11a47e6a4a22f8c51311cde, State: Initialized, Role: FOLLOWER
I20260926 14:50:38.838961 20592 consensus_queue.cc:260] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:38.839067 20237 raft_consensus.cc:3037] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:38.839196 20592 ts_tablet_manager.cc:1428] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:50:38.839320 20064 catalog_manager.cc:4841] TS 44d126fab62e44f998597b16f5ba6a65 (127.12.68.194:33691): tablet 8ff84e3e9da34fe6ac8f2b10ce7383e5 (table test-table [id=8e617e163c604fc9a480a7b5fcc59ec1]) successfully deleted
I20260926 14:50:38.839560 20601 raft_consensus.cc:2781] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:38.839641 20601 raft_consensus.cc:686] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [term 1 LEADER]: Becoming Leader. State: Replica: 44d126fab62e44f998597b16f5ba6a65, State: Running, Role: LEADER
I20260926 14:50:38.839740 20601 consensus_queue.cc:237] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
W20260926 14:50:38.839836 20424 tablet.cc:2369] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:38.840019 20237 raft_consensus.cc:2454] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d126fab62e44f998597b16f5ba6a65 in term 1.
I20260926 14:50:38.841490 20079 catalog_manager.cc:5495] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d126fab62e44f998597b16f5ba6a65 (127.12.68.194). New cstate: current_term: 1 leader_uuid: "44d126fab62e44f998597b16f5ba6a65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } health_report { overall_health: UNKNOWN } } }
Sep 26 14:50:38 dist-test-slave-24mn krb5kdc[20035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434237, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.254@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
I20260926 14:50:38.873744 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.865277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:33117 (local address 127.0.0.1:35880)
0926 14:50:38.865538 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.865552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.865652 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.865857 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.865859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.866358 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.866463 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.866470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.867208 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.867211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.868131 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.868140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.868542 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.868545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.868673 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.869169 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.869182 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.870674 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:38.872246 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.872255 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.872261 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.872535 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.872773 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.872775 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.872777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.872863 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.873156 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:38.873163 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:38.873279 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 14:50:38.873391 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 14:50:38.873533 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":31,"thread_start_us":79,"threads_started":1}
I20260926 14:50:38.873760 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.865344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.12.68.254:33117)
0926 14:50:38.865459 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.865464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.865615 (+ 151us) server_negotiation.cc:407] Connection header received
0926 14:50:38.865726 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.865728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.865762 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.865822 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.866610 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.867112 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.868232 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.868437 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.868529 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.870794 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:38.870797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:38.870799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:38.870819 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:38.872152 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.872628 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.872631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.872632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.872701 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.872931 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.872932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.872934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.873075 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:38.873143 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.873421 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.873606 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":46}
I20260926 14:50:38.877305 20079 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:35880:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 14:50:38.877959 20079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:38.882238 20216 tablet_service.cc:1467] Processing CreateTablet for tablet d1aaf9d1ef564498b0c4e32295b3c0bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0b136214dd34aad8ced4e33f08596a9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:38.882627 20216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1aaf9d1ef564498b0c4e32295b3c0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:38.882506 20357 tablet_service.cc:1467] Processing CreateTablet for tablet d1aaf9d1ef564498b0c4e32295b3c0bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0b136214dd34aad8ced4e33f08596a9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:38.882818 20357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1aaf9d1ef564498b0c4e32295b3c0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:38.884351 20497 tablet_service.cc:1467] Processing CreateTablet for tablet d1aaf9d1ef564498b0c4e32295b3c0bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0b136214dd34aad8ced4e33f08596a9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:38.884670 20497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1aaf9d1ef564498b0c4e32295b3c0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:38.885736 20592 tablet_bootstrap.cc:492] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde: Bootstrap starting.
I20260926 14:50:38.886650 20592 tablet_bootstrap.cc:654] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:38.887259 20591 tablet_bootstrap.cc:492] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e: Bootstrap starting.
I20260926 14:50:38.888151 20591 tablet_bootstrap.cc:654] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:38.888640 20593 tablet_bootstrap.cc:492] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65: Bootstrap starting.
I20260926 14:50:38.889515 20593 tablet_bootstrap.cc:654] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:38.890023 20592 tablet_bootstrap.cc:492] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde: No bootstrap required, opened a new log
I20260926 14:50:38.890105 20592 ts_tablet_manager.cc:1397] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260926 14:50:38.890523 20592 raft_consensus.cc:348] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:38.890640 20591 tablet_bootstrap.cc:492] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e: No bootstrap required, opened a new log
I20260926 14:50:38.890638 20592 raft_consensus.cc:374] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:38.890672 20592 raft_consensus.cc:729] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39bfa3c4f11a47e6a4a22f8c51311cde, State: Initialized, Role: FOLLOWER
I20260926 14:50:38.890702 20591 ts_tablet_manager.cc:1397] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 14:50:38.890789 20592 consensus_queue.cc:260] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:38.891058 20592 ts_tablet_manager.cc:1428] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:38.891094 20591 raft_consensus.cc:348] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:38.891193 20591 raft_consensus.cc:374] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:38.891223 20591 raft_consensus.cc:729] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8725b41178594da893bc86987ca15e1e, State: Initialized, Role: FOLLOWER
I20260926 14:50:38.891286 20599 tablet_replica.cc:439] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } }
I20260926 14:50:38.891326 20591 consensus_queue.cc:260] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:38.891404 20599 tablet_replica.cc:442] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:38.891574 20591 ts_tablet_manager.cc:1428] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:38.891541 20596 tablet_replica.cc:439] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } }
I20260926 14:50:38.891674 20596 tablet_replica.cc:442] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:38.893777 20593 tablet_bootstrap.cc:492] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65: No bootstrap required, opened a new log
I20260926 14:50:38.893857 20593 ts_tablet_manager.cc:1397] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 14:50:38.894220 20593 raft_consensus.cc:348] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:38.894309 20593 raft_consensus.cc:374] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:38.894337 20593 raft_consensus.cc:729] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d126fab62e44f998597b16f5ba6a65, State: Initialized, Role: FOLLOWER
I20260926 14:50:38.894448 20593 consensus_queue.cc:260] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:38.894676 20593 ts_tablet_manager.cc:1428] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:38.894660 20610 tablet_replica.cc:439] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } }
I20260926 14:50:38.894769 20610 tablet_replica.cc:442] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:39.185611 20596 raft_consensus.cc:482] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:39.185742 20596 raft_consensus.cc:504] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.186714 20596 leader_election.cc:288] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193:44621), 44d126fab62e44f998597b16f5ba6a65 (127.12.68.194:33691)
I20260926 14:50:39.190340 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.187071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:39035 (local address 127.12.68.193:44621)
0926 14:50:39.187188 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.187193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.187344 (+ 151us) server_negotiation.cc:407] Connection header received
0926 14:50:39.187446 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.187449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.187495 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.187550 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:39.187825 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.188340 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.189416 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.189803 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.189892 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.189942 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:39.190146 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 14:50:39.190217 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:50:39.190333 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.186943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44621 (local address 127.12.68.195:39035)
0926 14:50:39.187256 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:39.187268 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:39.187358 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:39.187578 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:39.187581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:39.187596 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:39.187723 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.187730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.188438 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.188442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.189294 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.189303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.189911 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.189915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.190017 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.190054 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 14:50:39.190108 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 14:50:39.190169 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260926 14:50:39.190367 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.187061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:50841 (local address 127.12.68.194:33691)
0926 14:50:39.187181 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.187186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.187396 (+ 210us) server_negotiation.cc:407] Connection header received
0926 14:50:39.187517 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.187519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.187569 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.187638 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:39.187866 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.188370 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.189444 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.189834 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.189937 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.189989 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:39.190199 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 14:50:39.190269 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:50:39.190356 20632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.186971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:33691 (local address 127.12.68.195:50841)
0926 14:50:39.187301 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:39.187313 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:39.187411 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:39.187662 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:39.187665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:39.187680 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:39.187764 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.187769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.188486 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.188489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.189319 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.189327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.189943 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.189947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.190061 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.190078 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 14:50:39.190168 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 14:50:39.190226 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":77,"thread_start_us":77,"threads_started":1}
I20260926 14:50:39.190763 20237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1aaf9d1ef564498b0c4e32295b3c0bc" candidate_uuid: "8725b41178594da893bc86987ca15e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" is_pre_election: true
I20260926 14:50:39.190829 20377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1aaf9d1ef564498b0c4e32295b3c0bc" candidate_uuid: "8725b41178594da893bc86987ca15e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d126fab62e44f998597b16f5ba6a65" is_pre_election: true
I20260926 14:50:39.190898 20237 raft_consensus.cc:2454] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8725b41178594da893bc86987ca15e1e in term 0.
I20260926 14:50:39.191056 20377 raft_consensus.cc:2454] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8725b41178594da893bc86987ca15e1e in term 0.
I20260926 14:50:39.191205 20450 leader_election.cc:302] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39bfa3c4f11a47e6a4a22f8c51311cde, 8725b41178594da893bc86987ca15e1e; no voters:
I20260926 14:50:39.191448 20596 raft_consensus.cc:2781] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:39.191504 20596 raft_consensus.cc:482] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:39.191541 20596 raft_consensus.cc:3037] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:39.192528 20596 raft_consensus.cc:504] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.192828 20596 leader_election.cc:288] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [CANDIDATE]: Term 1 election: Requested vote from peers 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193:44621), 44d126fab62e44f998597b16f5ba6a65 (127.12.68.194:33691)
I20260926 14:50:39.193117 20237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1aaf9d1ef564498b0c4e32295b3c0bc" candidate_uuid: "8725b41178594da893bc86987ca15e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde"
I20260926 14:50:39.193228 20237 raft_consensus.cc:3037] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:39.193256 20377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1aaf9d1ef564498b0c4e32295b3c0bc" candidate_uuid: "8725b41178594da893bc86987ca15e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d126fab62e44f998597b16f5ba6a65"
I20260926 14:50:39.193365 20377 raft_consensus.cc:3037] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:39.194012 20237 raft_consensus.cc:2454] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8725b41178594da893bc86987ca15e1e in term 1.
I20260926 14:50:39.194264 20450 leader_election.cc:302] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39bfa3c4f11a47e6a4a22f8c51311cde, 8725b41178594da893bc86987ca15e1e; no voters:
I20260926 14:50:39.194290 20377 raft_consensus.cc:2454] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8725b41178594da893bc86987ca15e1e in term 1.
I20260926 14:50:39.194437 20596 raft_consensus.cc:2781] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:39.194614 20596 raft_consensus.cc:686] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [term 1 LEADER]: Becoming Leader. State: Replica: 8725b41178594da893bc86987ca15e1e, State: Running, Role: LEADER
I20260926 14:50:39.194849 20596 consensus_queue.cc:237] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.196058 20633 tablet_replica.cc:439] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e: TxnStatusTablet state changed. Reason: New leader 8725b41178594da893bc86987ca15e1e. Latest consensus state: current_term: 1 leader_uuid: "8725b41178594da893bc86987ca15e1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } }
I20260926 14:50:39.196167 20633 tablet_replica.cc:442] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:39.196406 20635 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 14:50:39.196555 20635 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 14:50:39.197237 20079 catalog_manager.cc:5495] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8725b41178594da893bc86987ca15e1e (127.12.68.195). New cstate: current_term: 1 leader_uuid: "8725b41178594da893bc86987ca15e1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } health_report { overall_health: HEALTHY } } }
I20260926 14:50:39.257591 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.253778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43662 (local address 127.12.68.195:46871)
0926 14:50:39.254028 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.254031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.254043 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:39.254106 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.254108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.254156 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.254242 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:39.254480 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.254999 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.256173 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.256396 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.256470 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.256813 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:39.257236 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:39.257328 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:39.257415 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 14:50:39.257464 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":55,"threads_started":1}
I20260926 14:50:39.257618 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.253651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:46871 (local address 127.0.0.1:43662)
0926 14:50:39.253812 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:39.253828 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:39.253929 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:39.254264 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:39.254267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:39.254275 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:39.254384 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.254389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.255086 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.255089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.256070 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.256081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.256484 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.256487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.256581 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.256643 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:39.257323 (+ 680us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:39.257334 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 14:50:39.257395 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:50:39.257474 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 14:50:39.260411 20237 raft_consensus.cc:1261] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Refusing update from remote peer 8725b41178594da893bc86987ca15e1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:39.260543 20377 raft_consensus.cc:1261] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Refusing update from remote peer 8725b41178594da893bc86987ca15e1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:39.260923 20633 consensus_queue.cc:1035] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:39.261289 20596 consensus_queue.cc:1035] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:39.262655 20599 tablet_replica.cc:439] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde: TxnStatusTablet state changed. Reason: New leader 8725b41178594da893bc86987ca15e1e. Latest consensus state: current_term: 1 leader_uuid: "8725b41178594da893bc86987ca15e1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } }
I20260926 14:50:39.262758 20599 tablet_replica.cc:442] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:39.262841 20610 tablet_replica.cc:439] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65: TxnStatusTablet state changed. Reason: New leader 8725b41178594da893bc86987ca15e1e. Latest consensus state: current_term: 1 leader_uuid: "8725b41178594da893bc86987ca15e1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } }
I20260926 14:50:39.262948 20610 tablet_replica.cc:442] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:39.263958 20596 tablet_replica.cc:439] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8725b41178594da893bc86987ca15e1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } }
I20260926 14:50:39.264055 20596 tablet_replica.cc:442] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:39.265439 20599 tablet_replica.cc:439] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8725b41178594da893bc86987ca15e1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } }
I20260926 14:50:39.265537 20599 tablet_replica.cc:442] T d1aaf9d1ef564498b0c4e32295b3c0bc P 39bfa3c4f11a47e6a4a22f8c51311cde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:39.268342 20633 tablet_replica.cc:439] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8725b41178594da893bc86987ca15e1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } }
I20260926 14:50:39.268437 20633 tablet_replica.cc:442] T d1aaf9d1ef564498b0c4e32295b3c0bc P 8725b41178594da893bc86987ca15e1e: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:39.269390 20610 tablet_replica.cc:439] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8725b41178594da893bc86987ca15e1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } } }
I20260926 14:50:39.269505 20610 tablet_replica.cc:442] T d1aaf9d1ef564498b0c4e32295b3c0bc P 44d126fab62e44f998597b16f5ba6a65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:39.277853 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.274760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:53007 (local address 127.12.68.195:46871)
0926 14:50:39.274909 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.274912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.274965 (+ 53us) server_negotiation.cc:407] Connection header received
0926 14:50:39.275092 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.275094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.275130 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.275183 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:39.275382 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.275843 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.276997 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.277196 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.277261 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.277532 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:39.277588 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:39.277632 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:39.277721 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 14:50:39.277761 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 14:50:39.277948 20643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.274663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:46871 (local address 127.12.68.194:53007)
0926 14:50:39.274901 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:39.274915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:39.275012 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:39.275201 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:39.275204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:39.275211 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:39.275304 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.275309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.275958 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.275961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.276892 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.276904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.277291 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.277295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.277381 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.277427 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:39.277650 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:39.277655 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:39.277716 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:50:39.277799 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
I20260926 14:50:39.283325 20237 raft_consensus.cc:1261] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Refusing update from remote peer 44d126fab62e44f998597b16f5ba6a65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:39.283325 20517 raft_consensus.cc:1261] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Refusing update from remote peer 44d126fab62e44f998597b16f5ba6a65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:39.283661 20601 consensus_queue.cc:1035] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:39.283847 20610 consensus_queue.cc:1035] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:39.303736 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.300540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:58989 (local address 127.12.68.194:33691)
0926 14:50:39.300657 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.300662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.300920 (+ 258us) server_negotiation.cc:407] Connection header received
0926 14:50:39.301071 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.301074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.301122 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.301201 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:39.301442 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.301955 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.302823 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.303028 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.303107 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.303347 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:39.303417 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:39.303472 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:39.303562 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 14:50:39.303608 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:50:39.303783 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.300458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:33691 (local address 127.12.68.195:58989)
0926 14:50:39.300837 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:39.300851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:39.300971 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:39.301213 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:39.301216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:39.301223 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:39.301353 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.301358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.302040 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.302043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.302720 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.302729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.303113 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.303116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.303202 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.303239 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:39.303494 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:39.303500 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:50:39.303554 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 14:50:39.303624 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":88,"threads_started":1}
I20260926 14:50:39.323767 20079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:39.323920 20079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:39.325755 20079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553: Sending DeleteTablet for 3 replicas of tablet 7f810f3ddd5542e9af95bb51ad2a90d8
I20260926 14:50:39.326188 20216 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f810f3ddd5542e9af95bb51ad2a90d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:39 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:33458
I20260926 14:50:39.326432 20650 tablet_replica.cc:330] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde: stopping tablet replica
I20260926 14:50:39.326406 20357 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f810f3ddd5542e9af95bb51ad2a90d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:39 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:37850
I20260926 14:50:39.326545 20616 tablet_replica.cc:330] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65: stopping tablet replica
I20260926 14:50:39.326660 20616 raft_consensus.cc:2229] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:39.326668 20650 raft_consensus.cc:2229] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:39.326807 20616 raft_consensus.cc:2258] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:39.326911 20650 raft_consensus.cc:2258] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:39.327433 20616 ts_tablet_manager.cc:1905] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:39.329751 20650 ts_tablet_manager.cc:1905] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:39.330026 20616 ts_tablet_manager.cc:1918] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:39.330101 20616 log.cc:1196] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/7f810f3ddd5542e9af95bb51ad2a90d8
I20260926 14:50:39.330322 20616 ts_tablet_manager.cc:1939] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 44d126fab62e44f998597b16f5ba6a65: Deleting consensus metadata
I20260926 14:50:39.330859 20497 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f810f3ddd5542e9af95bb51ad2a90d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:39 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:45588
I20260926 14:50:39.330940 20064 catalog_manager.cc:4841] TS 44d126fab62e44f998597b16f5ba6a65 (127.12.68.194:33691): tablet 7f810f3ddd5542e9af95bb51ad2a90d8 (table test-table [id=1a847960d12d4b8fa2b422f3d8a9ec05]) successfully deleted
I20260926 14:50:39.332029 20652 tablet_replica.cc:330] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e: stopping tablet replica
I20260926 14:50:39.332019 20650 ts_tablet_manager.cc:1918] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:39.332083 20650 log.cc:1196] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/wal/wals/7f810f3ddd5542e9af95bb51ad2a90d8
I20260926 14:50:39.332245 20652 raft_consensus.cc:2229] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:39.332320 20650 ts_tablet_manager.cc:1939] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 39bfa3c4f11a47e6a4a22f8c51311cde: Deleting consensus metadata
I20260926 14:50:39.332427 20652 raft_consensus.cc:2258] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:39.332872 20066 catalog_manager.cc:4841] TS 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193:44621): tablet 7f810f3ddd5542e9af95bb51ad2a90d8 (table test-table [id=1a847960d12d4b8fa2b422f3d8a9ec05]) successfully deleted
I20260926 14:50:39.333153 20652 ts_tablet_manager.cc:1905] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:39.335482 20652 ts_tablet_manager.cc:1918] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:39.335546 20652 log.cc:1196] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/7f810f3ddd5542e9af95bb51ad2a90d8
I20260926 14:50:39.335743 20652 ts_tablet_manager.cc:1939] T 7f810f3ddd5542e9af95bb51ad2a90d8 P 8725b41178594da893bc86987ca15e1e: Deleting consensus metadata
I20260926 14:50:39.336326 20064 catalog_manager.cc:4841] TS 8725b41178594da893bc86987ca15e1e (127.12.68.195:46871): tablet 7f810f3ddd5542e9af95bb51ad2a90d8 (table test-table [id=1a847960d12d4b8fa2b422f3d8a9ec05]) successfully deleted
Sep 26 14:50:39 dist-test-slave-24mn krb5kdc[20035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:50:39 dist-test-slave-24mn krb5kdc[20035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:39 dist-test-slave-24mn krb5kdc[20035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:50:39.355852 20129 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:39.349721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.12.68.254:33117)
0926 14:50:39.349848 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.349853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.349983 (+ 130us) server_negotiation.cc:407] Connection header received
0926 14:50:39.350110 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.350112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.350153 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.350215 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:39.350974 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.351470 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.352650 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.352855 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.352924 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.355739 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35886: BlockingRecv error: recv got EOF from 127.0.0.1:35886 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 14:50:39 dist-test-slave-24mn krb5kdc[20035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM
I20260926 14:50:39.364362 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.357273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.12.68.254:33117)
0926 14:50:39.357412 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.357415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.357555 (+ 140us) server_negotiation.cc:407] Connection header received
0926 14:50:39.357657 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.357660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.357698 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.357755 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:39.358280 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.358758 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.359478 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.359700 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.359777 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.361799 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:39.361802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:39.361805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:39.361829 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:39.362963 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:39.363319 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:39.363323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:39.363325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:39.363397 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:39.363625 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:39.363629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:39.363631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:39.363788 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:39.363913 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:39.364127 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 14:50:39.364224 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 14:50:39.366428 20079 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:39.366798 20079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:39.370946 20357 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce6804d343e452393761e2b81b4046e (DEFAULT_TABLE table=test-table [id=6543924c411b45689fdacf8b64901421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:39.370982 20497 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce6804d343e452393761e2b81b4046e (DEFAULT_TABLE table=test-table [id=6543924c411b45689fdacf8b64901421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:39.371011 20216 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce6804d343e452393761e2b81b4046e (DEFAULT_TABLE table=test-table [id=6543924c411b45689fdacf8b64901421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:39.371214 20357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ce6804d343e452393761e2b81b4046e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:39.371214 20497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ce6804d343e452393761e2b81b4046e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:39.371271 20216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ce6804d343e452393761e2b81b4046e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:39.373070 20592 tablet_bootstrap.cc:492] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde: Bootstrap starting.
I20260926 14:50:39.373411 20591 tablet_bootstrap.cc:492] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e: Bootstrap starting.
I20260926 14:50:39.373430 20593 tablet_bootstrap.cc:492] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65: Bootstrap starting.
I20260926 14:50:39.373944 20592 tablet_bootstrap.cc:654] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:39.374053 20593 tablet_bootstrap.cc:654] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:39.374159 20591 tablet_bootstrap.cc:654] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:39.375133 20593 tablet_bootstrap.cc:492] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65: No bootstrap required, opened a new log
I20260926 14:50:39.375221 20593 ts_tablet_manager.cc:1397] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:39.375274 20591 tablet_bootstrap.cc:492] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e: No bootstrap required, opened a new log
I20260926 14:50:39.375296 20592 tablet_bootstrap.cc:492] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde: No bootstrap required, opened a new log
I20260926 14:50:39.375350 20591 ts_tablet_manager.cc:1397] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:39.375372 20592 ts_tablet_manager.cc:1397] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:39.375612 20593 raft_consensus.cc:348] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.375725 20593 raft_consensus.cc:374] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:39.375692 20591 raft_consensus.cc:348] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.375758 20593 raft_consensus.cc:729] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d126fab62e44f998597b16f5ba6a65, State: Initialized, Role: FOLLOWER
I20260926 14:50:39.375790 20591 raft_consensus.cc:374] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:39.375741 20592 raft_consensus.cc:348] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.375820 20591 raft_consensus.cc:729] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8725b41178594da893bc86987ca15e1e, State: Initialized, Role: FOLLOWER
I20260926 14:50:39.375829 20592 raft_consensus.cc:374] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:39.375857 20592 raft_consensus.cc:729] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39bfa3c4f11a47e6a4a22f8c51311cde, State: Initialized, Role: FOLLOWER
I20260926 14:50:39.375870 20593 consensus_queue.cc:260] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.375950 20591 consensus_queue.cc:260] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.375984 20592 consensus_queue.cc:260] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.376129 20593 ts_tablet_manager.cc:1428] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:39.376175 20591 ts_tablet_manager.cc:1428] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:39.376216 20592 ts_tablet_manager.cc:1428] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:39.609274 20649 raft_consensus.cc:482] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:39.609402 20649 raft_consensus.cc:504] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.609791 20649 leader_election.cc:288] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193:44621), 44d126fab62e44f998597b16f5ba6a65 (127.12.68.194:33691)
I20260926 14:50:39.610183 20377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce6804d343e452393761e2b81b4046e" candidate_uuid: "8725b41178594da893bc86987ca15e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d126fab62e44f998597b16f5ba6a65" is_pre_election: true
I20260926 14:50:39.610205 20237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce6804d343e452393761e2b81b4046e" candidate_uuid: "8725b41178594da893bc86987ca15e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" is_pre_election: true
I20260926 14:50:39.610321 20377 raft_consensus.cc:2454] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8725b41178594da893bc86987ca15e1e in term 0.
I20260926 14:50:39.610344 20237 raft_consensus.cc:2454] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8725b41178594da893bc86987ca15e1e in term 0.
I20260926 14:50:39.610612 20448 leader_election.cc:302] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d126fab62e44f998597b16f5ba6a65, 8725b41178594da893bc86987ca15e1e; no voters:
I20260926 14:50:39.610775 20649 raft_consensus.cc:2781] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:39.610832 20649 raft_consensus.cc:482] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:39.610872 20649 raft_consensus.cc:3037] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:39.611810 20649 raft_consensus.cc:504] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.612118 20649 leader_election.cc:288] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [CANDIDATE]: Term 1 election: Requested vote from peers 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193:44621), 44d126fab62e44f998597b16f5ba6a65 (127.12.68.194:33691)
I20260926 14:50:39.612408 20237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce6804d343e452393761e2b81b4046e" candidate_uuid: "8725b41178594da893bc86987ca15e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde"
I20260926 14:50:39.612443 20377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce6804d343e452393761e2b81b4046e" candidate_uuid: "8725b41178594da893bc86987ca15e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d126fab62e44f998597b16f5ba6a65"
I20260926 14:50:39.612527 20377 raft_consensus.cc:3037] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:39.612536 20237 raft_consensus.cc:3037] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:39.613337 20377 raft_consensus.cc:2454] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8725b41178594da893bc86987ca15e1e in term 1.
I20260926 14:50:39.613394 20237 raft_consensus.cc:2454] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8725b41178594da893bc86987ca15e1e in term 1.
I20260926 14:50:39.613577 20448 leader_election.cc:302] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d126fab62e44f998597b16f5ba6a65, 8725b41178594da893bc86987ca15e1e; no voters:
I20260926 14:50:39.613722 20649 raft_consensus.cc:2781] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:39.613806 20649 raft_consensus.cc:686] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [term 1 LEADER]: Becoming Leader. State: Replica: 8725b41178594da893bc86987ca15e1e, State: Running, Role: LEADER
I20260926 14:50:39.613936 20649 consensus_queue.cc:237] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.615434 20080 catalog_manager.cc:5495] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8725b41178594da893bc86987ca15e1e (127.12.68.195). New cstate: current_term: 1 leader_uuid: "8725b41178594da893bc86987ca15e1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } health_report { overall_health: HEALTHY } } }
I20260926 14:50:39.668970 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.666081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.12.68.195:46871)
0926 14:50:39.666198 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.666201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.666214 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:39.666294 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.666296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.666336 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.666396 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:39.666674 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.667168 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.668045 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.668275 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.668356 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.668577 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:39.668646 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:39.668694 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:39.668820 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 14:50:39.668864 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:50:39.671063 20237 raft_consensus.cc:1261] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Refusing update from remote peer 8725b41178594da893bc86987ca15e1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:39.671063 20377 raft_consensus.cc:1261] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Refusing update from remote peer 8725b41178594da893bc86987ca15e1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:39.671475 20596 consensus_queue.cc:1035] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:39.671651 20649 consensus_queue.cc:1035] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:39.679924 20079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:39.680060 20079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:39.681830 20079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553: Sending DeleteTablet for 3 replicas of tablet 6ce6804d343e452393761e2b81b4046e
I20260926 14:50:39.682281 20216 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ce6804d343e452393761e2b81b4046e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:39 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:33458
I20260926 14:50:39.682396 20357 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ce6804d343e452393761e2b81b4046e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:39 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:37850
I20260926 14:50:39.682458 20650 tablet_replica.cc:330] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde: stopping tablet replica
I20260926 14:50:39.682548 20616 tablet_replica.cc:330] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65: stopping tablet replica
I20260926 14:50:39.682560 20650 raft_consensus.cc:2229] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:39.682657 20616 raft_consensus.cc:2229] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:39.682667 20650 raft_consensus.cc:2258] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:39.682757 20616 raft_consensus.cc:2258] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:39.683128 20497 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ce6804d343e452393761e2b81b4046e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:39 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:45588
I20260926 14:50:39.683171 20079 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 14:50:39.683295 20652 tablet_replica.cc:330] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e: stopping tablet replica
I20260926 14:50:39.683384 20652 raft_consensus.cc:2229] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [term 1 LEADER]: Raft consensus shutting down.
W20260926 14:50:39.683529 20079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:39.683650 20652 raft_consensus.cc:2258] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:39.683830 20650 ts_tablet_manager.cc:1905] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:39.684412 20652 ts_tablet_manager.cc:1905] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:39.686179 20650 ts_tablet_manager.cc:1918] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:39.686239 20650 log.cc:1196] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/wal/wals/6ce6804d343e452393761e2b81b4046e
I20260926 14:50:39.686452 20650 ts_tablet_manager.cc:1939] T 6ce6804d343e452393761e2b81b4046e P 39bfa3c4f11a47e6a4a22f8c51311cde: Deleting consensus metadata
I20260926 14:50:39.686684 20652 ts_tablet_manager.cc:1918] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:39.686746 20652 log.cc:1196] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/6ce6804d343e452393761e2b81b4046e
I20260926 14:50:39.686967 20652 ts_tablet_manager.cc:1939] T 6ce6804d343e452393761e2b81b4046e P 8725b41178594da893bc86987ca15e1e: Deleting consensus metadata
I20260926 14:50:39.687268 20616 ts_tablet_manager.cc:1905] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:39.687597 20066 catalog_manager.cc:4841] TS 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193:44621): tablet 6ce6804d343e452393761e2b81b4046e (table test-table [id=6543924c411b45689fdacf8b64901421]) successfully deleted
I20260926 14:50:39.687876 20064 catalog_manager.cc:4841] TS 8725b41178594da893bc86987ca15e1e (127.12.68.195:46871): tablet 6ce6804d343e452393761e2b81b4046e (table test-table [id=6543924c411b45689fdacf8b64901421]) successfully deleted
I20260926 14:50:39.688372 20357 tablet_service.cc:1467] Processing CreateTablet for tablet c6d9e785d6c64e9898fd033eba792d5d (DEFAULT_TABLE table=test-table [id=6b792988b403489983dc8a71947c469e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:39.688609 20357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d9e785d6c64e9898fd033eba792d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:39.689459 20616 ts_tablet_manager.cc:1918] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:39.689518 20616 log.cc:1196] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/6ce6804d343e452393761e2b81b4046e
I20260926 14:50:39.689746 20616 ts_tablet_manager.cc:1939] T 6ce6804d343e452393761e2b81b4046e P 44d126fab62e44f998597b16f5ba6a65: Deleting consensus metadata
I20260926 14:50:39.690265 20593 tablet_bootstrap.cc:492] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65: Bootstrap starting.
I20260926 14:50:39.690953 20064 catalog_manager.cc:4841] TS 44d126fab62e44f998597b16f5ba6a65 (127.12.68.194:33691): tablet 6ce6804d343e452393761e2b81b4046e (table test-table [id=6543924c411b45689fdacf8b64901421]) successfully deleted
I20260926 14:50:39.691004 20593 tablet_bootstrap.cc:654] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:39.691329 20497 tablet_service.cc:1467] Processing CreateTablet for tablet c6d9e785d6c64e9898fd033eba792d5d (DEFAULT_TABLE table=test-table [id=6b792988b403489983dc8a71947c469e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:39.691540 20497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d9e785d6c64e9898fd033eba792d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:39.691524 20216 tablet_service.cc:1467] Processing CreateTablet for tablet c6d9e785d6c64e9898fd033eba792d5d (DEFAULT_TABLE table=test-table [id=6b792988b403489983dc8a71947c469e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:39.691727 20216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d9e785d6c64e9898fd033eba792d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:39.692512 20593 tablet_bootstrap.cc:492] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65: No bootstrap required, opened a new log
I20260926 14:50:39.692576 20593 ts_tablet_manager.cc:1397] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:50:39.692915 20593 raft_consensus.cc:348] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.693004 20593 raft_consensus.cc:374] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:39.693032 20593 raft_consensus.cc:729] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d126fab62e44f998597b16f5ba6a65, State: Initialized, Role: FOLLOWER
I20260926 14:50:39.693132 20593 consensus_queue.cc:260] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.693352 20593 ts_tablet_manager.cc:1428] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:39.693953 20591 tablet_bootstrap.cc:492] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e: Bootstrap starting.
I20260926 14:50:39.694212 20592 tablet_bootstrap.cc:492] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde: Bootstrap starting.
I20260926 14:50:39.694686 20591 tablet_bootstrap.cc:654] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:39.694985 20592 tablet_bootstrap.cc:654] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:39.696010 20592 tablet_bootstrap.cc:492] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde: No bootstrap required, opened a new log
I20260926 14:50:39.696086 20592 ts_tablet_manager.cc:1397] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:39.696465 20592 raft_consensus.cc:348] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.696528 20591 tablet_bootstrap.cc:492] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e: No bootstrap required, opened a new log
I20260926 14:50:39.696570 20592 raft_consensus.cc:374] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:39.696596 20591 ts_tablet_manager.cc:1397] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 14:50:39.696625 20592 raft_consensus.cc:729] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39bfa3c4f11a47e6a4a22f8c51311cde, State: Initialized, Role: FOLLOWER
I20260926 14:50:39.696730 20592 consensus_queue.cc:260] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.696976 20592 ts_tablet_manager.cc:1428] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:39.696950 20591 raft_consensus.cc:348] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.697130 20591 raft_consensus.cc:374] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:39.697176 20591 raft_consensus.cc:729] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8725b41178594da893bc86987ca15e1e, State: Initialized, Role: FOLLOWER
I20260926 14:50:39.697289 20591 consensus_queue.cc:260] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.697523 20591 ts_tablet_manager.cc:1428] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:39.859485 20599 raft_consensus.cc:482] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:39.859639 20599 raft_consensus.cc:504] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.860694 20599 leader_election.cc:288] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d126fab62e44f998597b16f5ba6a65 (127.12.68.194:33691), 8725b41178594da893bc86987ca15e1e (127.12.68.195:46871)
I20260926 14:50:39.864965 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.860986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:46293 (local address 127.12.68.194:33691)
0926 14:50:39.861210 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.861215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.861296 (+ 81us) server_negotiation.cc:407] Connection header received
0926 14:50:39.861495 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.861499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.861549 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.861659 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:39.862006 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.862719 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.863986 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.864383 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.864498 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.864553 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:39.864779 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 14:50:39.864849 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":83,"threads_started":1}
I20260926 14:50:39.864976 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.861379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:39041 (local address 127.12.68.195:46871)
0926 14:50:39.861498 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.861501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.861676 (+ 175us) server_negotiation.cc:407] Connection header received
0926 14:50:39.861828 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.861830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.861876 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.861946 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:39.862233 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.862871 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.864006 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.864395 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.864495 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.864568 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:39.864803 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 14:50:39.864884 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:50:39.865000 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.860899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:33691 (local address 127.12.68.193:46293)
0926 14:50:39.861196 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:39.861212 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:39.861337 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:39.861668 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:39.861671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:39.861716 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:39.861849 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.861856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.862834 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.862837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.863839 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.863849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.864499 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.864503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.864667 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.864686 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 14:50:39.864750 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:50:39.864820 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":68,"spinlock_wait_cycles":29312,"thread_start_us":109,"threads_started":1}
I20260926 14:50:39.865320 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.861326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:46871 (local address 127.12.68.193:39041)
0926 14:50:39.861609 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:39.861622 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:39.861724 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:39.862003 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:39.862006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:39.862025 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:39.862132 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.862138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.862953 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.862956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.863912 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.863923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.864510 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.864514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.864672 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.864694 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:50:39.865118 (+ 424us) client_negotiation.cc:240] Negotiation successful
0926 14:50:39.865191 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":29,"spinlock_wait_cycles":9344,"thread_start_us":49,"threads_started":1}
I20260926 14:50:39.865353 20377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d9e785d6c64e9898fd033eba792d5d" candidate_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d126fab62e44f998597b16f5ba6a65" is_pre_election: true
I20260926 14:50:39.865499 20377 raft_consensus.cc:2454] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39bfa3c4f11a47e6a4a22f8c51311cde in term 0.
I20260926 14:50:39.865839 20168 leader_election.cc:302] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39bfa3c4f11a47e6a4a22f8c51311cde, 44d126fab62e44f998597b16f5ba6a65; no voters:
I20260926 14:50:39.865919 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d9e785d6c64e9898fd033eba792d5d" candidate_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8725b41178594da893bc86987ca15e1e" is_pre_election: true
I20260926 14:50:39.866039 20517 raft_consensus.cc:2454] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39bfa3c4f11a47e6a4a22f8c51311cde in term 0.
I20260926 14:50:39.866060 20599 raft_consensus.cc:2781] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:39.866109 20599 raft_consensus.cc:482] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:39.866153 20599 raft_consensus.cc:3037] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:39.867069 20599 raft_consensus.cc:504] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.867326 20599 leader_election.cc:288] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [CANDIDATE]: Term 1 election: Requested vote from peers 44d126fab62e44f998597b16f5ba6a65 (127.12.68.194:33691), 8725b41178594da893bc86987ca15e1e (127.12.68.195:46871)
I20260926 14:50:39.867626 20377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d9e785d6c64e9898fd033eba792d5d" candidate_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d126fab62e44f998597b16f5ba6a65"
I20260926 14:50:39.867681 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d9e785d6c64e9898fd033eba792d5d" candidate_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8725b41178594da893bc86987ca15e1e"
I20260926 14:50:39.867746 20377 raft_consensus.cc:3037] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:39.867763 20517 raft_consensus.cc:3037] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:39.868607 20377 raft_consensus.cc:2454] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39bfa3c4f11a47e6a4a22f8c51311cde in term 1.
I20260926 14:50:39.868624 20517 raft_consensus.cc:2454] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39bfa3c4f11a47e6a4a22f8c51311cde in term 1.
I20260926 14:50:39.868839 20168 leader_election.cc:302] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39bfa3c4f11a47e6a4a22f8c51311cde, 44d126fab62e44f998597b16f5ba6a65; no voters:
I20260926 14:50:39.868971 20599 raft_consensus.cc:2781] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:39.869127 20599 raft_consensus.cc:686] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 LEADER]: Becoming Leader. State: Replica: 39bfa3c4f11a47e6a4a22f8c51311cde, State: Running, Role: LEADER
I20260926 14:50:39.869372 20599 consensus_queue.cc:237] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } }
I20260926 14:50:39.871781 20079 catalog_manager.cc:5495] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde reported cstate change: term changed from 0 to 1, leader changed from <none> to 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193). New cstate: current_term: 1 leader_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bfa3c4f11a47e6a4a22f8c51311cde" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 44621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:39.890925 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.887368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46646 (local address 127.12.68.193:44621)
0926 14:50:39.887637 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.887645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.887658 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:39.887728 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.887731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.887787 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.887917 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:39.888158 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.888683 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.889475 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.889706 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.889787 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.890012 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:39.890439 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:39.890537 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:39.890774 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 14:50:39.890818 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":123,"threads_started":1}
I20260926 14:50:39.899914 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.896856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:50197 (local address 127.12.68.195:46871)
0926 14:50:39.897005 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.897009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.897106 (+ 97us) server_negotiation.cc:407] Connection header received
0926 14:50:39.897236 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.897239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.897277 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.897344 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:39.897555 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.898066 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.898926 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.899172 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.899252 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.899510 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:39.899575 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:39.899630 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:39.899719 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 14:50:39.899761 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 14:50:39.899945 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.896737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:46871 (local address 127.12.68.193:50197)
0926 14:50:39.897028 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:39.897041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:39.897137 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:39.897353 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:39.897355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:39.897362 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:39.897463 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.897469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.898176 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.898180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.898819 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.898827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.899263 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.899267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.899355 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.899392 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:39.899648 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:39.899653 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:39.899709 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:50:39.899781 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
I20260926 14:50:39.904853 20517 raft_consensus.cc:1261] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Refusing update from remote peer 39bfa3c4f11a47e6a4a22f8c51311cde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:39.904910 20377 raft_consensus.cc:1261] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Refusing update from remote peer 39bfa3c4f11a47e6a4a22f8c51311cde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:39.905246 20599 consensus_queue.cc:1035] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "8725b41178594da893bc86987ca15e1e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 46871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:39.905405 20678 consensus_queue.cc:1035] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d126fab62e44f998597b16f5ba6a65" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 33691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:39.920915 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.917979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:36625 (local address 127.12.68.193:44621)
0926 14:50:39.918077 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.918081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.918261 (+ 180us) server_negotiation.cc:407] Connection header received
0926 14:50:39.918338 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.918340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.918374 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.918427 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:39.918573 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.919038 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.919927 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.920243 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.920324 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.920566 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:39.920645 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:39.920709 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:39.920786 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 14:50:39.920826 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:50:39.920955 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.917927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44621 (local address 127.12.68.195:36625)
0926 14:50:39.918191 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:39.918204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:39.918279 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:39.918437 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:39.918439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:39.918444 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:39.918519 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.918523 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.919105 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.919107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.919802 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.919811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.920343 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.920346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.920421 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.920451 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:39.920712 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:39.920718 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:50:39.920777 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 14:50:39.920851 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":57,"threads_started":1}
I20260926 14:50:39.940717 20080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:39.940851 20080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:39.942744 20080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58540ac4b0d4475eb9a19e933dc55553: Sending DeleteTablet for 3 replicas of tablet c6d9e785d6c64e9898fd033eba792d5d
I20260926 14:50:39.943267 20216 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d9e785d6c64e9898fd033eba792d5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:39 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:33458
I20260926 14:50:39.943362 20497 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d9e785d6c64e9898fd033eba792d5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:39 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:45588
I20260926 14:50:39.943447 20650 tablet_replica.cc:330] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde: stopping tablet replica
I20260926 14:50:39.943533 20652 tablet_replica.cc:330] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e: stopping tablet replica
I20260926 14:50:39.943562 20650 raft_consensus.cc:2229] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:39.943646 20652 raft_consensus.cc:2229] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:39.943744 20652 raft_consensus.cc:2258] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:39.943833 20650 raft_consensus.cc:2258] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:39.944510 20652 ts_tablet_manager.cc:1905] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:39.944542 20650 ts_tablet_manager.cc:1905] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:39.946336 20652 ts_tablet_manager.cc:1918] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:39.946398 20652 log.cc:1196] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/c6d9e785d6c64e9898fd033eba792d5d
I20260926 14:50:39.946601 20652 ts_tablet_manager.cc:1939] T c6d9e785d6c64e9898fd033eba792d5d P 8725b41178594da893bc86987ca15e1e: Deleting consensus metadata
I20260926 14:50:39.946841 20650 ts_tablet_manager.cc:1918] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:39.946918 20650 log.cc:1196] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-0/wal/wals/c6d9e785d6c64e9898fd033eba792d5d
I20260926 14:50:39.947162 20650 ts_tablet_manager.cc:1939] T c6d9e785d6c64e9898fd033eba792d5d P 39bfa3c4f11a47e6a4a22f8c51311cde: Deleting consensus metadata
I20260926 14:50:39.947206 20064 catalog_manager.cc:4841] TS 8725b41178594da893bc86987ca15e1e (127.12.68.195:46871): tablet c6d9e785d6c64e9898fd033eba792d5d (table test-table [id=6b792988b403489983dc8a71947c469e]) successfully deleted
I20260926 14:50:39.947535 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 20152
I20260926 14:50:39.949295 20357 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d9e785d6c64e9898fd033eba792d5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:39 UTC) from {username='oryx', principal='oryx/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:37850
I20260926 14:50:39.949460 20616 tablet_replica.cc:330] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65: stopping tablet replica
I20260926 14:50:39.949563 20616 raft_consensus.cc:2229] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:39.949677 20616 raft_consensus.cc:2258] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:39.950349 20616 ts_tablet_manager.cc:1905] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:39.952581 20616 ts_tablet_manager.cc:1918] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:39.952658 20616 log.cc:1196] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/c6d9e785d6c64e9898fd033eba792d5d
I20260926 14:50:39.952874 20616 ts_tablet_manager.cc:1939] T c6d9e785d6c64e9898fd033eba792d5d P 44d126fab62e44f998597b16f5ba6a65: Deleting consensus metadata
I20260926 14:50:39.953437 20064 catalog_manager.cc:4841] TS 44d126fab62e44f998597b16f5ba6a65 (127.12.68.194:33691): tablet c6d9e785d6c64e9898fd033eba792d5d (table test-table [id=6b792988b403489983dc8a71947c469e]) successfully deleted
W20260926 14:50:39.954545 20066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:44621 (error 108)
I20260926 14:50:39.954741 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 20292
I20260926 14:50:39.960161 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 20432
I20260926 14:50:39.960271 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.955414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:44621 (local address 127.0.0.1:46662)
0926 14:50:39.960121 (+ 4707us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:39.960182 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:44621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4630,"thread_start_us":52,"threads_started":1}
W20260926 14:50:39.961516 20066 catalog_manager.cc:4584] TS 39bfa3c4f11a47e6a4a22f8c51311cde (127.12.68.193:44621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6d9e785d6c64e9898fd033eba792d5d: Network error: Client connection negotiation failed: client connection to 127.12.68.193:44621: connect: Connection refused (error 111)
I20260926 14:50:39.965580 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 20048
2026-09-26T14:50:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3302 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:40 dist-test-slave-24mn krb5kdc[20695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:40 dist-test-slave-24mn krb5kdc[20695](info): set up 2 sockets
Sep 26 14:50:40 dist-test-slave-24mn krb5kdc[20695](info): commencing operation
krb5kdc: starting...
W20260926 14:50:40.912209 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:50:40 dist-test-slave-24mn krb5kdc[20695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:50:41.020354 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:34663
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:34663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:41.121783 20708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:41.122004 20708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:41.122041 20708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:41.125909 20708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:41.125954 20708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:41.125972 20708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:41.125988 20708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:41.126015 20708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:41.131034 20708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:34663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:34663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:41.131964 20708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:41.132886 20708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:41.140667 20713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:41.140687 20716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:41.140728 20708 server_base.cc:1034] running on GCE node
W20260926 14:50:41.140674 20714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:41.141288 20708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:41.142259 20708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:41.143407 20708 hybrid_clock.cc:648] HybridClock initialized: now 1790434241143399 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T14:50:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1182 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:41 dist-test-slave-24mn krb5kdc[20721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:41 dist-test-slave-24mn krb5kdc[20721](info): set up 2 sockets
Sep 26 14:50:41 dist-test-slave-24mn krb5kdc[20721](info): commencing operation
krb5kdc: starting...
W20260926 14:50:42.092286 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:50:42 dist-test-slave-24mn krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:50:42.202066 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:36535
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:36535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:42.305097 20734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:42.305331 20734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:42.305384 20734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:42.309302 20734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:42.309352 20734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:42.309377 20734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:42.309399 20734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:42.309423 20734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:42.314417 20734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:36535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:36535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:42.315335 20734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:42.316284 20734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:42.324015 20740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:42.324031 20742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:42.324067 20734 server_base.cc:1034] running on GCE node
W20260926 14:50:42.324015 20739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:42.324810 20734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:42.325789 20734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:42.326954 20734 hybrid_clock.cc:648] HybridClock initialized: now 1790434242326937 us; error 33 us; skew 500 ppm
Sep 26 14:50:42 dist-test-slave-24mn krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434242, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:42.334424 20734 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:50:42.335227 20734 webserver.cc:460] Webserver started at http://127.12.68.254:33543/ using document root <none> and password file <none>
I20260926 14:50:42.335724 20734 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:42.335785 20734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:42.335990 20734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:42.337910 20734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "af0c910f37034b4b92e192fcb2d9b7f6"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "b11dc8df176e5f6eefd1ff7a28732491"
server_key_iv: "2db010c78c98324c0b73d0f2baae6b7a"
server_key_version: "encryptionkey@0"
I20260926 14:50:42.338409 20734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "af0c910f37034b4b92e192fcb2d9b7f6"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "b11dc8df176e5f6eefd1ff7a28732491"
server_key_iv: "2db010c78c98324c0b73d0f2baae6b7a"
server_key_version: "encryptionkey@0"
I20260926 14:50:42.341770 20734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:50:42.343869 20749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:42.345027 20734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:42.345139 20734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "af0c910f37034b4b92e192fcb2d9b7f6"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "b11dc8df176e5f6eefd1ff7a28732491"
server_key_iv: "2db010c78c98324c0b73d0f2baae6b7a"
server_key_version: "encryptionkey@0"
I20260926 14:50:42.345222 20734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:42.368045 20734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:42.370605 20734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:42.370762 20734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:42.376789 20734 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:36535
I20260926 14:50:42.376803 20801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:36535 every 8 connection(s)
I20260926 14:50:42.377643 20734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:42.380137 20802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:42.385260 20802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6: Bootstrap starting.
I20260926 14:50:42.387199 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 20734
I20260926 14:50:42.387333 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:42.387539 20802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:42.387562 12563 external_mini_cluster.cc:1428] Setting key 9b37e2f53d447544c5fbd55002590ebb
I20260926 14:50:42.388295 20802 log.cc:824] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:42.390141 20802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6: No bootstrap required, opened a new log
I20260926 14:50:42.394004 20802 raft_consensus.cc:348] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36535 } }
I20260926 14:50:42.394239 20802 raft_consensus.cc:374] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:42.394299 20802 raft_consensus.cc:729] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0c910f37034b4b92e192fcb2d9b7f6, State: Initialized, Role: FOLLOWER
I20260926 14:50:42.394791 20802 consensus_queue.cc:260] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36535 } }
I20260926 14:50:42.394959 20802 raft_consensus.cc:388] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:42.395023 20802 raft_consensus.cc:482] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:42.395118 20802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 14:50:42 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:42.396026 20802 raft_consensus.cc:504] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36535 } }
I20260926 14:50:42.396374 20802 leader_election.cc:302] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af0c910f37034b4b92e192fcb2d9b7f6; no voters:
I20260926 14:50:42.396648 20802 leader_election.cc:288] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:42.396971 20807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:42.397214 20807 raft_consensus.cc:686] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 1 LEADER]: Becoming Leader. State: Replica: af0c910f37034b4b92e192fcb2d9b7f6, State: Running, Role: LEADER
I20260926 14:50:42.397526 20807 consensus_queue.cc:237] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36535 } }
I20260926 14:50:42.397905 20802 sys_catalog.cc:564] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:42.398979 20808 sys_catalog.cc:455] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36535 } } }
I20260926 14:50:42.399094 20808 sys_catalog.cc:458] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:42.399453 20809 sys_catalog.cc:455] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af0c910f37034b4b92e192fcb2d9b7f6. Latest consensus state: current_term: 1 leader_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36535 } } }
I20260926 14:50:42.399617 20809 sys_catalog.cc:458] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:42.400226 20816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:42.401404 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:42.388545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.12.68.254:36535)
0926 14:50:42.388908 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:42.388922 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:42.388954 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:50:42.389550 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:42.389566 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:42.389873 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:42.390141 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:42.390969 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.391961 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.393116 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.393547 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.393655 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:42.396333 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:42.396338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:42.396351 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:42.396375 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:42.398274 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:42.398759 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:42.398766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:42.398773 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:42.398917 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:42.399248 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:42.399252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:42.399254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:42.399755 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:42.399960 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:42.400362 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 14:50:42.400650 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":90,"threads_started":1}
I20260926 14:50:42.402995 20816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:42.407807 20816 catalog_manager.cc:1347] Generated new cluster ID: 3bd58d64e00f403aa55e591e46473d97
I20260926 14:50:42.407866 20816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:42.417977 20816 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:42.418803 20816 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:42.424351 20816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6: Generated new TSK 0
I20260926 14:50:42.424938 20816 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:50:42.456444 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:36535
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:42.560021 20828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:42.560238 20828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:42.560277 20828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:42.564191 20828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:42.564239 20828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:42.564301 20828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:42.569289 20828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:42.570171 20828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:42.571089 20828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:42.579439 20836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:42.579453 20833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:42.579438 20834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:42.580214 20828 server_base.cc:1034] running on GCE node
I20260926 14:50:42.580578 20828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:42.581178 20828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:42.582324 20828 hybrid_clock.cc:648] HybridClock initialized: now 1790434242582309 us; error 25 us; skew 500 ppm
Sep 26 14:50:42 dist-test-slave-24mn krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434242, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:42.584810 20828 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:50:42.585579 20828 webserver.cc:460] Webserver started at http://127.12.68.193:40615/ using document root <none> and password file <none>
I20260926 14:50:42.586058 20828 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:42.586128 20828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:42.586318 20828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:42.588207 20828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "2cb1a4bfe9e64175ad6bd8311c712858"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "c053eb7ec4644f62d490b05b38381b48"
server_key_iv: "2609a1432fefbbb2fca68702f468e4e8"
server_key_version: "encryptionkey@0"
I20260926 14:50:42.588707 20828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "2cb1a4bfe9e64175ad6bd8311c712858"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "c053eb7ec4644f62d490b05b38381b48"
server_key_iv: "2609a1432fefbbb2fca68702f468e4e8"
server_key_version: "encryptionkey@0"
I20260926 14:50:42.591981 20828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:42.594125 20843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:42.595206 20828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:42.595305 20828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "2cb1a4bfe9e64175ad6bd8311c712858"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "c053eb7ec4644f62d490b05b38381b48"
server_key_iv: "2609a1432fefbbb2fca68702f468e4e8"
server_key_version: "encryptionkey@0"
I20260926 14:50:42.595386 20828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:42.609176 20828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:42.611739 20828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:42.611910 20828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:42.612419 20828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:42.613150 20828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:42.613194 20828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:42.613241 20828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:42.613269 20828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:42.620972 20828 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:36763
I20260926 14:50:42.621001 20956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:36763 every 8 connection(s)
I20260926 14:50:42.621744 20828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:50:42 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434242, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:42.631515 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 20828
I20260926 14:50:42.631645 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:42.632000 12563 external_mini_cluster.cc:1428] Setting key ea79c154ee4e6548feba9a7112123162
I20260926 14:50:42.633837 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:42.623325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:34265 (local address 127.12.68.254:36535)
0926 14:50:42.623433 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:42.623436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:42.624392 (+ 956us) server_negotiation.cc:407] Connection header received
0926 14:50:42.625220 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:42.625223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:42.625278 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:42.625344 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:42.626398 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.626939 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.628018 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.628249 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.628335 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:42.630435 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:42.630438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:42.630440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:42.630462 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:42.631936 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:42.632411 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:42.632413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:42.632414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:42.632496 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:42.632829 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:42.632832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:42.632833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:42.632995 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:42.633072 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:42.633586 (+ 514us) server_negotiation.cc:299] Negotiation successful
0926 14:50:42.633701 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:50:42.634539 20959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:42.623634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:34265)
0926 14:50:42.624246 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:42.624282 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:42.625032 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:42.625511 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:42.625517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:42.625827 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:42.626263 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:42.626276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.627051 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.627053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:42.627907 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:42.627916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.628375 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.628379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:42.628550 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:42.629028 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:42.629043 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:42.630289 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:42.632040 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:42.632046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:42.632060 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:42.632313 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:42.632587 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:42.632589 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:42.632590 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:42.632746 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:42.633089 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:42.633095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:42.633368 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 14:50:42.633517 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 14:50:42.633761 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":100,"threads_started":1}
I20260926 14:50:42.635638 20957 heartbeater.cc:347] Connected to a master server at 127.12.68.254:36535
I20260926 14:50:42.635857 20957 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:42.636363 20957 heartbeater.cc:511] Master 127.12.68.254:36535 requested a full tablet report, sending...
I20260926 14:50:42.637744 20766 ts_manager.cc:194] Registered new tserver with Master: 2cb1a4bfe9e64175ad6bd8311c712858 (127.12.68.193:36763)
I20260926 14:50:42.638809 20766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:34265
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:50:42.656742 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:36535
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:42.760936 20962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:42.761166 20962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:42.761205 20962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:42.765079 20962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:42.765131 20962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:42.765184 20962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:42.770176 20962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:42.771095 20962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:42.772066 20962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:42.780795 20970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:42.780884 20962 server_base.cc:1034] running on GCE node
W20260926 14:50:42.780807 20967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:42.780812 20968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:42.781463 20962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:42.782055 20962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:42.783213 20962 hybrid_clock.cc:648] HybridClock initialized: now 1790434242783187 us; error 37 us; skew 500 ppm
Sep 26 14:50:42 dist-test-slave-24mn krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434242, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:42.785667 20962 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:50:42.786463 20962 webserver.cc:460] Webserver started at http://127.12.68.194:35131/ using document root <none> and password file <none>
I20260926 14:50:42.786958 20962 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:42.787019 20962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:42.787191 20962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:42.789083 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "9ef964470e8a48a494e8e4713053f613"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "d223237ee8434864593048f741b5e35a"
server_key_iv: "ecd0d7e33c054065ef2152d6a46ed6ef"
server_key_version: "encryptionkey@0"
I20260926 14:50:42.789570 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "9ef964470e8a48a494e8e4713053f613"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "d223237ee8434864593048f741b5e35a"
server_key_iv: "ecd0d7e33c054065ef2152d6a46ed6ef"
server_key_version: "encryptionkey@0"
I20260926 14:50:42.793143 20962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 14:50:42.795424 20977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:42.796868 20962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:42.796976 20962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "9ef964470e8a48a494e8e4713053f613"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "d223237ee8434864593048f741b5e35a"
server_key_iv: "ecd0d7e33c054065ef2152d6a46ed6ef"
server_key_version: "encryptionkey@0"
I20260926 14:50:42.797091 20962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:42.814533 20962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:42.817112 20962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:42.817265 20962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:42.817777 20962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:42.818528 20962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:42.818570 20962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:42.818614 20962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:42.818642 20962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:42.826278 20962 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:43071
I20260926 14:50:42.826333 21090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:43071 every 8 connection(s)
I20260926 14:50:42.827064 20962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:42.832041 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 20962
I20260926 14:50:42.832182 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:42.832461 12563 external_mini_cluster.cc:1428] Setting key f8090954c269624e731a62dd6b9fc970
Sep 26 14:50:42 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434242, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:42.840386 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:42.828727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:52861 (local address 127.12.68.254:36535)
0926 14:50:42.828860 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:42.828864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:42.829876 (+ 1012us) server_negotiation.cc:407] Connection header received
0926 14:50:42.830656 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:42.830659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:42.830701 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:42.830770 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:42.831908 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.832672 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.834033 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.834326 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.834432 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:42.836890 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:42.836894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:42.836896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:42.836923 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:42.838523 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:42.838967 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:42.838971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:42.838973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:42.839070 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:42.839350 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:42.839353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:42.839355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:42.839567 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:42.839665 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:42.840114 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 14:50:42.840228 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:50:42.841070 21093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:42.829105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:52861)
0926 14:50:42.829735 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:42.829780 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:42.830483 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:42.830906 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:42.830919 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:42.831241 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:42.831758 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:42.831773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.832778 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.832782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:42.833920 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:42.833929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.834461 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.834463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:42.834609 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:42.835121 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:42.835138 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:42.836732 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:42.838628 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:42.838633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:42.838645 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:42.838876 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:42.839155 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:42.839157 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:42.839158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:42.839275 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:42.839659 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:42.839664 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:42.839862 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 14:50:42.840045 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 14:50:42.840277 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
I20260926 14:50:42.842061 21091 heartbeater.cc:347] Connected to a master server at 127.12.68.254:36535
I20260926 14:50:42.842267 21091 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:42.842694 21091 heartbeater.cc:511] Master 127.12.68.254:36535 requested a full tablet report, sending...
I20260926 14:50:42.843689 20766 ts_manager.cc:194] Registered new tserver with Master: 9ef964470e8a48a494e8e4713053f613 (127.12.68.194:43071)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
I20260926 14:50:42.844298 20766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:52861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:50:42.859117 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:36535
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:42.961714 21096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:42.961926 21096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:42.961964 21096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:42.965762 21096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:42.965807 21096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:42.965858 21096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:42.970769 21096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:42.971611 21096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:42.972469 21096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:42.980980 21104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:42.981002 21101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:42.981024 21102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:42.981102 21096 server_base.cc:1034] running on GCE node
I20260926 14:50:42.981494 21096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:42.982044 21096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:42.983199 21096 hybrid_clock.cc:648] HybridClock initialized: now 1790434242983182 us; error 29 us; skew 500 ppm
Sep 26 14:50:42 dist-test-slave-24mn krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434242, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:42.985617 21096 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:50:42.986411 21096 webserver.cc:460] Webserver started at http://127.12.68.195:42521/ using document root <none> and password file <none>
I20260926 14:50:42.986928 21096 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:42.986999 21096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:42.987186 21096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:42.989077 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "af2fd4db784547d88e2b4a861b1eafde"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "1a7a312fa8f300935065ecb1206a7805"
server_key_iv: "61575ec791a71108bb052f01c9b14201"
server_key_version: "encryptionkey@0"
I20260926 14:50:42.989564 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "af2fd4db784547d88e2b4a861b1eafde"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "1a7a312fa8f300935065ecb1206a7805"
server_key_iv: "61575ec791a71108bb052f01c9b14201"
server_key_version: "encryptionkey@0"
I20260926 14:50:42.992791 21096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:42.994865 21111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:42.995944 21096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:42.996044 21096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "af2fd4db784547d88e2b4a861b1eafde"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "1a7a312fa8f300935065ecb1206a7805"
server_key_iv: "61575ec791a71108bb052f01c9b14201"
server_key_version: "encryptionkey@0"
I20260926 14:50:42.996126 21096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:43.012020 21096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:43.014629 21096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:43.014786 21096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:43.015290 21096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:43.016067 21096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:43.016112 21096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:43.016161 21096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:43.016191 21096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:43.024046 21096 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:43921
I20260926 14:50:43.024092 21224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:43921 every 8 connection(s)
I20260926 14:50:43.024849 21096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434242, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:50:43.034070 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 21096
I20260926 14:50:43.034188 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:43.034441 12563 external_mini_cluster.cc:1428] Setting key 30501b0582d92ab97a4fc69b0a40522f
I20260926 14:50:43.037195 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.026473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:40875 (local address 127.12.68.254:36535)
0926 14:50:43.026612 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:43.026615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:43.027545 (+ 930us) server_negotiation.cc:407] Connection header received
0926 14:50:43.028349 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:43.028352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:43.028398 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:43.028478 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:43.029532 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.030054 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.031135 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.031364 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.031449 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.033714 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:43.033717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:43.033719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:43.033742 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:43.035295 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:43.035840 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:43.035843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:43.035845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:43.035955 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:43.036245 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:43.036248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:43.036250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:43.036467 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:43.036567 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:43.036995 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 14:50:43.037084 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 14:50:43.037899 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.026795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:40875)
0926 14:50:43.027397 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:43.027436 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:43.028190 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:43.028643 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:43.028650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:43.028969 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:43.029402 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.029415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.030141 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.030143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.031009 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.031017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.031528 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.031532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.031686 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.032218 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:43.032233 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:43.033562 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:43.035384 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:43.035390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:43.035403 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:43.035743 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:43.036034 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:43.036037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:43.036038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:43.036165 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:43.036598 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:43.036605 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:43.036810 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 14:50:43.036955 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 14:50:43.037178 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":116,"threads_started":1}
I20260926 14:50:43.038836 21225 heartbeater.cc:347] Connected to a master server at 127.12.68.254:36535
I20260926 14:50:43.039028 21225 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:43.039438 21225 heartbeater.cc:511] Master 127.12.68.254:36535 requested a full tablet report, sending...
I20260926 14:50:43.040365 20766 ts_manager.cc:194] Registered new tserver with Master: af2fd4db784547d88e2b4a861b1eafde (127.12.68.195:43921)
I20260926 14:50:43.040887 20766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:40875
I20260926 14:50:43.048028 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.193.keytab.
Entry for principal oryx/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.193.keytab.
WARNING: no policy specified for oryx/127.12.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.194.keytab.
Entry for principal oryx/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.194.keytab.
WARNING: no policy specified for oryx/127.12.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.195.keytab.
Entry for principal oryx/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.195.keytab.
I20260926 14:50:43.110023 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 20828
I20260926 14:50:43.114481 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 20962
I20260926 14:50:43.118778 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 21096
I20260926 14:50:43.122974 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 20734
I20260926 14:50:43.127801 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:36535
--webserver_interface=127.12.68.254
--webserver_port=33543
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:36535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:43.232004 21234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:43.232295 21234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:43.232358 21234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:43.236238 21234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:43.236282 21234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:43.236299 21234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:43.236315 21234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:43.236328 21234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:43.241374 21234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:36535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:36535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:43.242285 21234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:43.243201 21234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:43.251034 21239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:43.251075 21240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:43.251075 21242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:43.251276 21234 server_base.cc:1034] running on GCE node
I20260926 14:50:43.251793 21234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:43.252871 21234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:43.254026 21234 hybrid_clock.cc:648] HybridClock initialized: now 1790434243254008 us; error 30 us; skew 500 ppm
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434243, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:43.256527 21234 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:50:43.257303 21234 webserver.cc:460] Webserver started at http://127.12.68.254:33543/ using document root <none> and password file <none>
I20260926 14:50:43.257798 21234 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:43.257874 21234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:43.261392 21234 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 14:50:43.263314 21249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:43.264475 21234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260926 14:50:43.264590 21234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "af0c910f37034b4b92e192fcb2d9b7f6"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "b11dc8df176e5f6eefd1ff7a28732491"
server_key_iv: "2db010c78c98324c0b73d0f2baae6b7a"
server_key_version: "encryptionkey@0"
I20260926 14:50:43.265082 21234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:43.276731 21234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:43.279327 21234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:43.279493 21234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:43.285982 21234 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:36535
I20260926 14:50:43.286005 21301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:36535 every 8 connection(s)
I20260926 14:50:43.286936 21234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:43.290865 21302 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:50:43.292399 21302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6: Bootstrap starting.
I20260926 14:50:43.292845 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 21234
I20260926 14:50:43.293257 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:36763
--local_ip_for_outbound_sockets=127.12.68.193
--tserver_master_addrs=127.12.68.254:36535
--webserver_port=40615
--webserver_interface=127.12.68.193
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.68.193
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:50:43.308197 21302 log.cc:824] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:43.314288 21302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:50:43.314594 21302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6: Bootstrap complete.
I20260926 14:50:43.318437 21302 raft_consensus.cc:348] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36535 } }
I20260926 14:50:43.318825 21302 raft_consensus.cc:729] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0c910f37034b4b92e192fcb2d9b7f6, State: Initialized, Role: FOLLOWER
I20260926 14:50:43.319278 21302 consensus_queue.cc:260] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36535 } }
I20260926 14:50:43.319397 21302 raft_consensus.cc:388] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:43.319463 21302 raft_consensus.cc:482] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:43.319559 21302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:50:43.321362 21302 raft_consensus.cc:504] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36535 } }
I20260926 14:50:43.321671 21302 leader_election.cc:302] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af0c910f37034b4b92e192fcb2d9b7f6; no voters:
I20260926 14:50:43.321949 21302 leader_election.cc:288] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:50:43.322050 21306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:50:43.322228 21306 raft_consensus.cc:686] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [term 2 LEADER]: Becoming Leader. State: Replica: af0c910f37034b4b92e192fcb2d9b7f6, State: Running, Role: LEADER
I20260926 14:50:43.322568 21306 consensus_queue.cc:237] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36535 } }
I20260926 14:50:43.323098 21302 sys_catalog.cc:564] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:43.323925 21308 sys_catalog.cc:455] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af0c910f37034b4b92e192fcb2d9b7f6. Latest consensus state: current_term: 2 leader_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36535 } } }
I20260926 14:50:43.324051 21308 sys_catalog.cc:458] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:43.324306 21307 sys_catalog.cc:455] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0c910f37034b4b92e192fcb2d9b7f6" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36535 } } }
I20260926 14:50:43.324376 21307 sys_catalog.cc:458] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:43.324380 21315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:43.327437 21315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:43.328088 21315 catalog_manager.cc:1259] Loaded cluster ID: 3bd58d64e00f403aa55e591e46473d97
I20260926 14:50:43.328146 21315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:43.330487 21315 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:43.331130 21315 catalog_manager.cc:5879] T 00000000000000000000000000000000 P af0c910f37034b4b92e192fcb2d9b7f6: Loaded TSK: 0
I20260926 14:50:43.332325 21315 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:50:43.397817 21304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:43.398026 21304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:43.398063 21304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:43.401953 21304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:43.402001 21304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:43.402055 21304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:43.406947 21304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.193.keytab
--principal=oryx/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:36763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=40615
--tserver_master_addrs=127.12.68.254:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:43.407866 21304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:43.408823 21304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:43.418605 21329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:43.418686 21331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:43.418598 21328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:43.418854 21304 server_base.cc:1034] running on GCE node
I20260926 14:50:43.419284 21304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:43.419853 21304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:43.421031 21304 hybrid_clock.cc:648] HybridClock initialized: now 1790434243421006 us; error 44 us; skew 500 ppm
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434243, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:43.423514 21304 init.cc:377] Logged in from keytab as oryx/127.12.68.193@KRBTEST.COM (short username oryx)
I20260926 14:50:43.424443 21304 webserver.cc:460] Webserver started at http://127.12.68.193:40615/ using document root <none> and password file <none>
I20260926 14:50:43.424942 21304 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:43.425017 21304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:43.428588 21304 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:43.430528 21338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:43.431752 21304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:43.431850 21304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "2cb1a4bfe9e64175ad6bd8311c712858"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "c053eb7ec4644f62d490b05b38381b48"
server_key_iv: "2609a1432fefbbb2fca68702f468e4e8"
server_key_version: "encryptionkey@0"
I20260926 14:50:43.432359 21304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:43.449327 21304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:43.452208 21304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:43.452366 21304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:43.452883 21304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:43.453656 21304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:43.453702 21304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:43.453752 21304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:43.453776 21304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:43.461772 21304 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:36763
I20260926 14:50:43.461838 21451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:36763 every 8 connection(s)
I20260926 14:50:43.462672 21304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:43.468751 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 21304
I20260926 14:50:43.469264 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:43071
--local_ip_for_outbound_sockets=127.12.68.194
--tserver_master_addrs=127.12.68.254:36535
--webserver_port=35131
--webserver_interface=127.12.68.194
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.68.194
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:43.475734 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.464567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:43061)
0926 14:50:43.465200 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:43.465247 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:43.465972 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:43.466921 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:43.466927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:43.467254 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:43.467832 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.467851 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.469668 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.469672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.470906 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.470915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.471847 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.471850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.472029 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.472535 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:43.472552 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:43.474808 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":101,"threads_started":1}
W20260926 14:50:43.475725 21454 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:43.464438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:43061 (local address 127.12.68.254:36535)
0926 14:50:43.464808 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:43.464823 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:43.465355 (+ 532us) server_negotiation.cc:407] Connection header received
0926 14:50:43.466302 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:43.466312 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:43.466594 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:43.466851 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:43.468084 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.469520 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.471001 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.471706 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.471866 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.475026 (+ 3160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:43061: BlockingRecv error: recv got EOF from 127.12.68.193:43061 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":86,"threads_started":1}
W20260926 14:50:43.476204 21452 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:36535 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.68.254:36535: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:43.481207 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.476602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:49403)
0926 14:50:43.476806 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:43.476819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:43.476899 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:43.477109 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:43.477112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:43.477256 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:43.477339 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.477344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.478023 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.478026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.478556 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.478563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.478965 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.478968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.479059 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.479449 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:43.479462 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:43.481092 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146}
W20260926 14:50:43.481316 21454 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:43.476701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:49403 (local address 127.12.68.254:36535)
0926 14:50:43.476795 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:43.476799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:43.476871 (+ 72us) server_negotiation.cc:407] Connection header received
0926 14:50:43.476977 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:43.476980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:43.477024 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:43.477089 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:43.477436 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.477930 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.478656 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.478876 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.478955 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.481216 (+ 2261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:49403: BlockingRecv error: recv got EOF from 127.12.68.193:49403 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:43.486217 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.481777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:47833)
0926 14:50:43.481971 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:43.481983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:43.482067 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:43.482260 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:43.482262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:43.482392 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:43.482451 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.482456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.483102 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.483106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.483651 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.483660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.484114 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.484117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.484190 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.484564 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:43.484576 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:43.486105 (+ 1529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139}
W20260926 14:50:43.486312 21454 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:43.481871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:47833 (local address 127.12.68.254:36535)
0926 14:50:43.481971 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:43.481974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:43.482022 (+ 48us) server_negotiation.cc:407] Connection header received
0926 14:50:43.482138 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:43.482140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:43.482179 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:43.482239 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:43.482553 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.483011 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.483751 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.484022 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.484105 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.486216 (+ 2111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:47833: BlockingRecv error: recv got EOF from 127.12.68.193:47833 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 14:50:43.486496 21452 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:50:43.574743 21456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:43.574975 21456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:43.575016 21456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:43.578850 21456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:43.578902 21456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:43.578954 21456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:43.583863 21456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.194.keytab
--principal=oryx/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:43071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=35131
--tserver_master_addrs=127.12.68.254:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:43.584770 21456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:43.585651 21456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:43.594189 21461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:43.594189 21464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:43.594286 21462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:43.594457 21456 server_base.cc:1034] running on GCE node
I20260926 14:50:43.594913 21456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:43.595485 21456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:43.596632 21456 hybrid_clock.cc:648] HybridClock initialized: now 1790434243596611 us; error 32 us; skew 500 ppm
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434243, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:43.599030 21456 init.cc:377] Logged in from keytab as oryx/127.12.68.194@KRBTEST.COM (short username oryx)
I20260926 14:50:43.599776 21456 webserver.cc:460] Webserver started at http://127.12.68.194:35131/ using document root <none> and password file <none>
I20260926 14:50:43.600296 21456 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:43.600374 21456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:43.603742 21456 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:43.605571 21471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:43.606619 21456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:43.606717 21456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "9ef964470e8a48a494e8e4713053f613"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "d223237ee8434864593048f741b5e35a"
server_key_iv: "ecd0d7e33c054065ef2152d6a46ed6ef"
server_key_version: "encryptionkey@0"
I20260926 14:50:43.607195 21456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:43.620155 21456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:43.622596 21456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:43.622752 21456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:43.623262 21456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:43.624125 21456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:43.624177 21456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:43.624226 21456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:43.624254 21456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:43.632205 21456 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:43071
I20260926 14:50:43.632244 21584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:43071 every 8 connection(s)
I20260926 14:50:43.633116 21456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:43.635239 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 21456
I20260926 14:50:43.635718 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:43921
--local_ip_for_outbound_sockets=127.12.68.195
--tserver_master_addrs=127.12.68.254:36535
--webserver_port=42521
--webserver_interface=127.12.68.195
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.68.195
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:50:43.643335 21454 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:43.634695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:34913 (local address 127.12.68.254:36535)
0926 14:50:43.634815 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:43.634819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:43.635917 (+ 1098us) server_negotiation.cc:407] Connection header received
0926 14:50:43.636898 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:43.636902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:43.636960 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:43.637037 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:43.638363 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.638863 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.639968 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.640192 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.640279 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.643217 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:34913: BlockingRecv error: recv got EOF from 127.12.68.194:34913 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:50:43.643862 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.635042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:34913)
0926 14:50:43.635713 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:43.635784 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:43.636655 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:43.637241 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:43.637249 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:43.637690 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:43.638252 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.638266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.638978 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.638982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.639830 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.639841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.640323 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.640327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.640476 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.641023 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:43.641040 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:43.643084 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":139,"threads_started":1}
W20260926 14:50:43.644373 21585 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:36535 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.68.254:36535: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:43.649593 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.644777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:40029)
0926 14:50:43.644918 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:43.644932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:43.644993 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:43.645227 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:43.645229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:43.645349 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:43.645432 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.645436 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.646385 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.646388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.646912 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.646919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.647429 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.647431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.647498 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.647938 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:43.647952 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:43.649486 (+ 1534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86}
W20260926 14:50:43.649910 21454 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:43.644889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:40029 (local address 127.12.68.254:36535)
0926 14:50:43.645010 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:43.645014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:43.645028 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:50:43.645068 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:43.645071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:43.645115 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:43.645196 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:43.645526 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.646294 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.647001 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.647334 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.647423 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.649642 (+ 2219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:40029: BlockingRecv error: recv got EOF from 127.12.68.194:40029 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:43.654649 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.649983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:39163)
0926 14:50:43.650118 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:43.650129 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:43.650191 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:43.650423 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:43.650425 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:43.650529 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:43.650610 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.650614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.651554 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.651557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.652143 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.652152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.652664 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.652667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.652717 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.653103 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:43.653112 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:43.654568 (+ 1456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260926 14:50:43.654803 21454 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:43.650055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:39163 (local address 127.12.68.254:36535)
0926 14:50:43.650247 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:43.650251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:43.650264 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:43.650295 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:43.650298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:43.650331 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:43.650405 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:43.650698 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.651419 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.652239 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.652579 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.652671 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.654695 (+ 2024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:39163: BlockingRecv error: recv got EOF from 127.12.68.194:39163 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
W20260926 14:50:43.654899 21585 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:50:43.743010 21588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:43.743234 21588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:43.743274 21588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:43.748239 21588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:43.748306 21588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:43.748390 21588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:43.754678 21588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/krb5kdc/oryx_127.12.68.195.keytab
--principal=oryx/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:43921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=42521
--tserver_master_addrs=127.12.68.254:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:43.755635 21588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:43.756623 21588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:43.765213 21594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:43.765226 21595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:43.765226 21597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:43.765273 21588 server_base.cc:1034] running on GCE node
I20260926 14:50:43.765740 21588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:43.766300 21588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:43.767450 21588 hybrid_clock.cc:648] HybridClock initialized: now 1790434243767436 us; error 26 us; skew 500 ppm
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434243, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:43.769989 21588 init.cc:377] Logged in from keytab as oryx/127.12.68.195@KRBTEST.COM (short username oryx)
I20260926 14:50:43.770727 21588 webserver.cc:460] Webserver started at http://127.12.68.195:42521/ using document root <none> and password file <none>
I20260926 14:50:43.771245 21588 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:43.771318 21588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:43.774768 21588 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:43.776648 21604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:43.777724 21588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:43.777820 21588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "af2fd4db784547d88e2b4a861b1eafde"
format_stamp: "Formatted at 2026-09-26 14:50:42 on dist-test-slave-24mn"
server_key: "1a7a312fa8f300935065ecb1206a7805"
server_key_iv: "61575ec791a71108bb052f01c9b14201"
server_key_version: "encryptionkey@0"
I20260926 14:50:43.778298 21588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:43.796044 21588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:43.798480 21588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:43.798640 21588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:43.799127 21588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:43.799849 21588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:43.799919 21588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:43.799978 21588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:43.800007 21588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:43.808740 21588 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:43921
I20260926 14:50:43.808807 21717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:43921 every 8 connection(s)
I20260926 14:50:43.809911 21588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:43.811815 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 21588
I20260926 14:50:43.823149 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.812844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49010 (local address 127.12.68.254:36535)
0926 14:50:43.812952 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:43.812956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:43.813113 (+ 157us) server_negotiation.cc:407] Connection header received
0926 14:50:43.813309 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:43.813312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:43.813361 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:43.813418 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:43.814429 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.815234 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.816524 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.816874 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.816955 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.818720 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:43.818727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:43.818745 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:43.818778 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:43.820520 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:43.821139 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:43.821145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:43.821151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:43.821296 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:43.821945 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:43.821949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:43.821951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:43.822358 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:43.822549 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:43.822772 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 14:50:43.822999 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:50:43.825707 21454 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:43.812280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:54035 (local address 127.12.68.254:36535)
0926 14:50:43.812414 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:43.812419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:43.816436 (+ 4017us) server_negotiation.cc:407] Connection header received
0926 14:50:43.817503 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:43.817506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:43.817553 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:43.817619 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:43.818974 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.819573 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.821270 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.821562 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.821650 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.825565 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:54035: BlockingRecv error: recv got EOF from 127.12.68.195:54035 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:50:43.826924 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.812407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:54035)
0926 14:50:43.816307 (+ 3900us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:43.816343 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:43.817297 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:43.817800 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:43.817809 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:43.818243 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:43.818839 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.818857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.819808 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.819812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.821132 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.821143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.821731 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.821736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.821914 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.822560 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:43.822580 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:43.825999 (+ 3419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3721,"thread_start_us":134,"threads_started":1}
W20260926 14:50:43.827382 21718 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:36535 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.68.254:36535: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:43.833221 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.827750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:59371)
0926 14:50:43.827847 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:43.827860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:43.827949 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:43.828175 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:43.828178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:43.828308 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:43.828402 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.828409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.829324 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.829328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.830218 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.830229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.830769 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.830772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.830866 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.831285 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:43.831299 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:43.833077 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 14:50:43.833326 21454 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:43.827829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:59371 (local address 127.12.68.254:36535)
0926 14:50:43.827943 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:43.827947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:43.827960 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:43.828027 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:43.828031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:43.828072 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:43.828165 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:43.828553 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.829211 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.830343 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.830664 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.830744 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.833221 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:59371: BlockingRecv error: recv got EOF from 127.12.68.195:59371 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:43.839228 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.833950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:49155)
0926 14:50:43.834053 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:43.834066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:43.834145 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:43.834348 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:43.834350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:43.834499 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:43.834580 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.834586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.835464 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.835467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.836373 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.836384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.836850 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.836854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.837066 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.837432 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:43.837445 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:43.839112 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:50:43.839272 21454 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:43.834048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:49155 (local address 127.12.68.254:36535)
0926 14:50:43.834168 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:43.834172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:43.834185 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:43.834217 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:43.834219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:43.834253 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:43.834325 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:43.834673 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.835354 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.836473 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.836756 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.836846 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.839199 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:49155: BlockingRecv error: recv got EOF from 127.12.68.195:49155 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 14:50:43.839512 21718 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 14:50:44 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:44 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:44.491705 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:44.487000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:48547)
0926 14:50:44.487254 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:44.487267 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:44.487347 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:44.487546 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:44.487548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:44.487708 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:44.487811 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:44.487816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.488524 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.488526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:44.489070 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:44.489077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.489429 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.489431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:44.489494 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:44.489938 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:44.489949 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:44.491585 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
W20260926 14:50:44.491946 21723 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:44.487074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:48547 (local address 127.12.68.254:36535)
0926 14:50:44.487305 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:44.487308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:44.487320 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:44.487417 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:44.487420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:44.487460 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:44.487534 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:44.487914 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.488445 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.489154 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.489356 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.489422 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:44.491710 (+ 2288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:48547: BlockingRecv error: recv got EOF from 127.12.68.193:48547 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
Sep 26 14:50:44 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:44 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:44.660602 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:44.655459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:45833)
0926 14:50:44.655707 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:44.655722 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:44.655830 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:44.656078 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:44.656081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:44.656272 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:44.656379 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:44.656384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.657106 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.657109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:44.657680 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:44.657687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.658087 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.658090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:44.658186 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:44.658619 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:44.658629 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:44.660456 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
W20260926 14:50:44.660686 21723 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:44.655528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:45833 (local address 127.12.68.254:36535)
0926 14:50:44.655685 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:44.655689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:44.655788 (+ 99us) server_negotiation.cc:407] Connection header received
0926 14:50:44.655932 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:44.655935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:44.655983 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:44.656044 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:44.656482 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.657009 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.657781 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.657996 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.658077 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:44.660589 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:45833: BlockingRecv error: recv got EOF from 127.12.68.194:45833 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 14:50:44 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:44 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:44.844784 21725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:44.840050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:35919)
0926 14:50:44.840278 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:44.840292 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:44.840402 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:44.840617 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:44.840619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:44.840780 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:44.840869 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:44.840873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.841585 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.841589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:44.842165 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:44.842173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.842533 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.842536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:44.842598 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:44.843004 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:44.843014 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:44.844669 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
W20260926 14:50:44.844873 21723 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:44.840099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:35919 (local address 127.12.68.254:36535)
0926 14:50:44.840229 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:44.840234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:44.840349 (+ 115us) server_negotiation.cc:407] Connection header received
0926 14:50:44.840487 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:44.840489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:44.840535 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:44.840599 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:44.840948 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.841474 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.842253 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.842459 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.842523 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:44.844783 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:35919: BlockingRecv error: recv got EOF from 127.12.68.195:35919 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 14:50:45 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:45 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:45.496815 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:45.492313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:48805)
0926 14:50:45.492550 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:45.492562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:45.492658 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:45.492856 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:45.492858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:45.493013 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:45.493101 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:45.493106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.493756 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.493758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:45.494296 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:45.494302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.494653 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.494655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:45.494712 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:45.495125 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:45.495135 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:45.496706 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":78,"threads_started":1}
W20260926 14:50:45.496969 21727 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:45.492371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:48805 (local address 127.12.68.254:36535)
0926 14:50:45.492575 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:45.492579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:45.492608 (+ 29us) server_negotiation.cc:407] Connection header received
0926 14:50:45.492725 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:45.492727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:45.492767 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:45.492834 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:45.493178 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.493684 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.494370 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.494573 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.494635 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:45.496838 (+ 2203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:48805: BlockingRecv error: recv got EOF from 127.12.68.193:48805 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":52,"threads_started":1}
Sep 26 14:50:45 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:45 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:45.666401 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:45.661279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:49055)
0926 14:50:45.661517 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:45.661529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:45.661641 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:45.661865 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:45.661868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:45.662071 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:45.662161 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:45.662167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.662872 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.662876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:45.663443 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:45.663451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.663862 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.663866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:45.663973 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:45.664408 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:45.664418 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:45.666250 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
W20260926 14:50:45.666465 21727 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:45.661339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:49055 (local address 127.12.68.254:36535)
0926 14:50:45.661476 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:45.661480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:45.661591 (+ 111us) server_negotiation.cc:407] Connection header received
0926 14:50:45.661723 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:45.661726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:45.661772 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:45.661848 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:45.662265 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.662779 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.663547 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.663769 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.663851 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:45.666369 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:49055: BlockingRecv error: recv got EOF from 127.12.68.194:49055 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 14:50:45 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:45 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:45.849978 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:45.845451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:35735)
0926 14:50:45.845690 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:45.845702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:45.845789 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:45.845982 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:45.845984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:45.846146 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:45.846236 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:45.846241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.846906 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.846908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:45.847445 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:45.847451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.847802 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.847804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:45.847860 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:45.848296 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:45.848306 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:45.849853 (+ 1547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
W20260926 14:50:45.850060 21727 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:45.845573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:35735 (local address 127.12.68.254:36535)
0926 14:50:45.845689 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:45.845693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:45.845757 (+ 64us) server_negotiation.cc:407] Connection header received
0926 14:50:45.845857 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:45.845859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:45.845900 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:45.845960 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:45.846312 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.846826 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.847519 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.847730 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.847796 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:45.849969 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:35735: BlockingRecv error: recv got EOF from 127.12.68.195:35735 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 14:50:46 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:46 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:46.502162 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:46.497448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:54153)
0926 14:50:46.497715 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:46.497727 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:46.497811 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:46.498014 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:46.498016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:46.498188 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:46.498277 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:46.498282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.498948 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.498950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:46.499513 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:46.499520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.499958 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.499961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:46.500019 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:46.500438 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:46.500449 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:46.502034 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":67,"threads_started":1}
W20260926 14:50:46.502262 21731 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:46.497518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:54153 (local address 127.12.68.254:36535)
0926 14:50:46.497785 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:46.497788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:46.497801 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:46.497876 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:46.497878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:46.497920 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:46.497997 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:46.498357 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.498866 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.499616 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.499835 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.499934 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:46.502156 (+ 2222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:54153: BlockingRecv error: recv got EOF from 127.12.68.193:54153 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
Sep 26 14:50:46 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:46 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:46.671829 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:46.667057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:50513)
0926 14:50:46.667295 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:46.667306 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:46.667397 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:46.667625 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:46.667628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:46.667791 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:46.667916 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:46.667922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.668647 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.668650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:46.669243 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:46.669252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.669640 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.669642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:46.669703 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:46.670139 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:46.670150 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:46.671711 (+ 1561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
W20260926 14:50:46.671962 21731 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:46.667109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:50513 (local address 127.12.68.254:36535)
0926 14:50:46.667273 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:46.667278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:46.667366 (+ 88us) server_negotiation.cc:407] Connection header received
0926 14:50:46.667466 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:46.667468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:46.667533 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:46.667598 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:46.668001 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.668558 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.669328 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.669556 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.669628 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:46.671834 (+ 2206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:50513: BlockingRecv error: recv got EOF from 127.12.68.194:50513 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 14:50:46 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:46 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:46.855648 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:46.850665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:49137)
0926 14:50:46.850907 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:46.850920 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:46.851016 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:46.851215 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:46.851217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:46.851380 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:46.851503 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:46.851508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.852212 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.852214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:46.852804 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:46.852813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.853226 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.853230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:46.853306 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:46.853817 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:46.853832 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:46.855534 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
W20260926 14:50:46.855732 21731 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:46.850736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:49137 (local address 127.12.68.254:36535)
0926 14:50:46.850874 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:46.850878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:46.850982 (+ 104us) server_negotiation.cc:407] Connection header received
0926 14:50:46.851080 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:46.851082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:46.851125 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:46.851187 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:46.851590 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.852122 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.852914 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.853134 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.853207 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:46.855640 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:49137: BlockingRecv error: recv got EOF from 127.12.68.195:49137 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 14:50:47 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:47 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:47.511865 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:47.502817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:40485)
0926 14:50:47.503063 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:47.503076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:47.503169 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:47.503373 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:47.503374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:47.503548 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:47.503694 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:47.503703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.504632 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.504636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:47.505210 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:47.505219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.505615 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.505617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:47.505673 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:47.510022 (+ 4349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:47.510037 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:47.511754 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":68,"threads_started":1}
W20260926 14:50:47.512073 21735 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:47.502878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:40485 (local address 127.12.68.254:36535)
0926 14:50:47.503081 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:47.503084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:47.503123 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:50:47.503236 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:47.503239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:47.503278 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:47.503351 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:47.503798 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.504522 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.505304 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.505517 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.505589 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:47.511936 (+ 6347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:40485: BlockingRecv error: recv got EOF from 127.12.68.193:40485 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":61,"threads_started":1}
Sep 26 14:50:47 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:47 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:47.677450 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:47.672535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:55213)
0926 14:50:47.672788 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:47.672800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:47.672892 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:47.673149 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:47.673152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:47.673325 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:47.673421 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:47.673426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.674122 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.674125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:47.674700 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:47.674709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.675090 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.675093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:47.675159 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:47.675664 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:47.675676 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:47.677336 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
W20260926 14:50:47.677583 21735 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:47.672657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:55213 (local address 127.12.68.254:36535)
0926 14:50:47.672783 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:47.672787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:47.672858 (+ 71us) server_negotiation.cc:407] Connection header received
0926 14:50:47.672959 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:47.672961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:47.673002 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:47.673070 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:47.673535 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.674031 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.674785 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.674985 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.675049 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:47.677470 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:55213: BlockingRecv error: recv got EOF from 127.12.68.194:55213 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 14:50:47 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:47 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:47.861019 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:47.856324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:48765)
0926 14:50:47.856578 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:47.856590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:47.856679 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:47.856874 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:47.856876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:47.857033 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:47.857134 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:47.857139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.857874 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.857876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:47.858422 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:47.858430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.858792 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.858794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:47.858851 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:47.859303 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:47.859314 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:47.860905 (+ 1591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
W20260926 14:50:47.861115 21735 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:47.856396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:48765 (local address 127.12.68.254:36535)
0926 14:50:47.856514 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:47.856518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:47.856644 (+ 126us) server_negotiation.cc:407] Connection header received
0926 14:50:47.856742 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:47.856745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:47.856785 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:47.856848 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:47.857211 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.857787 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.858504 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.858719 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.858784 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:47.861018 (+ 2234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:48765: BlockingRecv error: recv got EOF from 127.12.68.195:48765 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 14:50:48 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:48 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:48.518023 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:48.512634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:38521)
0926 14:50:48.512976 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:48.512991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:48.513098 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:48.513403 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:48.513408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:48.513583 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:48.513678 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:48.513683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.514463 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.514466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:48.515058 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:48.515067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.515557 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.515560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:48.515635 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:48.516166 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:48.516178 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:48.517878 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":72,"threads_started":1}
W20260926 14:50:48.518132 21739 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:48.512688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:38521 (local address 127.12.68.254:36535)
0926 14:50:48.512911 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:48.512915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:48.513066 (+ 151us) server_negotiation.cc:407] Connection header received
0926 14:50:48.513204 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:48.513207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:48.513261 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:48.513347 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:48.513812 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.514343 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.515187 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.515434 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.515522 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:48.518008 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:38521: BlockingRecv error: recv got EOF from 127.12.68.193:38521 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
Sep 26 14:50:48 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:48 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:48.683254 21740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:48.678107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:58253)
0926 14:50:48.678347 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:48.678360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:48.678449 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:48.678664 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:48.678666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:48.678835 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:48.678926 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:48.678931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.679619 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.679621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:48.680210 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:48.680218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.680752 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.680755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:48.680840 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:48.681334 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:48.681346 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:48.683110 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":76,"threads_started":1}
W20260926 14:50:48.683332 21739 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:48.678238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:58253 (local address 127.12.68.254:36535)
0926 14:50:48.678370 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:48.678373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:48.678428 (+ 55us) server_negotiation.cc:407] Connection header received
0926 14:50:48.678532 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:48.678534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:48.678576 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:48.678639 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:48.679010 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.679534 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.680307 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.680542 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.680738 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:48.683236 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:58253: BlockingRecv error: recv got EOF from 127.12.68.194:58253 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 14:50:48 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:48 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:48.866365 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:48.861732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:41579)
0926 14:50:48.861964 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:48.861977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:48.862067 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:48.862274 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:48.862276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:48.862437 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:48.862527 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:48.862531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.863190 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.863192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:48.863731 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:48.863738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.864141 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.864144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:48.864201 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:48.864674 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:48.864685 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:48.866253 (+ 1568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
W20260926 14:50:48.866464 21739 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:48.861782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:41579 (local address 127.12.68.254:36535)
0926 14:50:48.861902 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:48.861906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:48.862037 (+ 131us) server_negotiation.cc:407] Connection header received
0926 14:50:48.862147 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:48.862150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:48.862190 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:48.862249 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:48.862612 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.863118 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.863807 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.864067 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.864138 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:48.866369 (+ 2231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:41579: BlockingRecv error: recv got EOF from 127.12.68.195:41579 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 14:50:49 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:49 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:49.524034 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:49.518841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:47513)
0926 14:50:49.519143 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:49.519157 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:49.519251 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:49.519474 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:49.519476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:49.519653 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:49.519744 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:49.519749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.520523 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.520526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:49.521094 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:49.521103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.521605 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.521608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:49.521672 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:49.522208 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:49.522219 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:49.523922 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
W20260926 14:50:49.524170 21743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:49.518940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:47513 (local address 127.12.68.254:36535)
0926 14:50:49.519190 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:49.519194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:49.519207 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:49.519318 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:49.519320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:49.519362 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:49.519440 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:49.519831 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.520404 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.521186 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.521507 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.521604 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:49.524032 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:47513: BlockingRecv error: recv got EOF from 127.12.68.193:47513 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
Sep 26 14:50:49 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:49 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:49.688748 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:49.684051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:56265)
0926 14:50:49.684284 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:49.684297 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:49.684393 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:49.684606 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:49.684608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:49.684779 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:49.684868 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:49.684872 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.685544 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.685546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:49.686090 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:49.686097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.686455 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.686457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:49.686512 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:49.686989 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:49.687001 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:49.688630 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
W20260926 14:50:49.688850 21743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:49.684111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:56265 (local address 127.12.68.254:36535)
0926 14:50:49.684261 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:49.684266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:49.684358 (+ 92us) server_negotiation.cc:407] Connection header received
0926 14:50:49.684461 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:49.684463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:49.684510 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:49.684570 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:49.684945 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.685468 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.686177 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.686382 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.686446 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:49.688743 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:56265: BlockingRecv error: recv got EOF from 127.12.68.194:56265 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 14:50:49 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:49 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:49.872133 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:49.867057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:50625)
0926 14:50:49.867335 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:49.867349 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:49.867447 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:49.867674 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:49.867676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:49.867845 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:49.867958 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:49.867963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.868693 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.868696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:49.869283 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:49.869292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.869669 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.869671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:49.869731 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:49.870161 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:49.870172 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:49.872022 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":87,"threads_started":1}
W20260926 14:50:49.872241 21743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:49.867126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:50625 (local address 127.12.68.254:36535)
0926 14:50:49.867255 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:49.867259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:49.867409 (+ 150us) server_negotiation.cc:407] Connection header received
0926 14:50:49.867527 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:49.867529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:49.867578 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:49.867640 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:49.868051 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.868607 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.869368 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.869588 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.869658 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:49.872131 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:50625: BlockingRecv error: recv got EOF from 127.12.68.195:50625 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 14:50:50 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:50 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:50.529932 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:50.524785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:47381)
0926 14:50:50.525088 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:50.525100 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:50.525197 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:50.525400 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:50.525402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:50.525568 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:50.525667 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:50.525673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.526436 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.526439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:50.527200 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:50.527211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.527628 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.527643 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:50.527699 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:50.528195 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:50.528209 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:50.529826 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
W20260926 14:50:50.530061 21747 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:50.524840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:47381 (local address 127.12.68.254:36535)
0926 14:50:50.525098 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:50.525102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:50.525164 (+ 62us) server_negotiation.cc:407] Connection header received
0926 14:50:50.525264 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:50.525266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:50.525308 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:50.525377 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:50.525749 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.526338 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.527310 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.527539 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.527617 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:50.529945 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:47381: BlockingRecv error: recv got EOF from 127.12.68.193:47381 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":89,"threads_started":1}
Sep 26 14:50:50 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:50 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:50.694051 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:50.689419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:58429)
0926 14:50:50.689706 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:50.689719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:50.689810 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:50.690008 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:50.690010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:50.690186 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:50.690272 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:50.690276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.690971 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.690973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:50.691521 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:50.691527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.691919 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.691922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:50.691983 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:50.692390 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:50.692400 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:50.693940 (+ 1540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":74,"threads_started":1}
W20260926 14:50:50.694185 21747 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:50.689505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:58429 (local address 127.12.68.254:36535)
0926 14:50:50.689660 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:50.689664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:50.689765 (+ 101us) server_negotiation.cc:407] Connection header received
0926 14:50:50.689883 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:50.689885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:50.689926 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:50.689979 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:50.690365 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.690895 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.691602 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.691813 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.691913 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:50.694070 (+ 2157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:58429: BlockingRecv error: recv got EOF from 127.12.68.194:58429 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 14:50:50 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:50 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:50.877682 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:50.872825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:35095)
0926 14:50:50.873091 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:50.873103 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:50.873194 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:50.873400 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:50.873402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:50.873592 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:50.873682 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:50.873687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.874343 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.874345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:50.874887 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:50.874893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.875344 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.875348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:50.875434 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:50.875963 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:50.875977 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:50.877574 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":63,"threads_started":1}
W20260926 14:50:50.877794 21747 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:50.872918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:35095 (local address 127.12.68.254:36535)
0926 14:50:50.873038 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:50.873041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:50.873160 (+ 119us) server_negotiation.cc:407] Connection header received
0926 14:50:50.873260 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:50.873263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:50.873305 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:50.873370 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:50.873760 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.874271 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.874987 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.875232 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.875315 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:50.877696 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:35095: BlockingRecv error: recv got EOF from 127.12.68.195:35095 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 14:50:51 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:51 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:51.535446 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:51.530675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:42613)
0926 14:50:51.530914 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:51.530927 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:51.531010 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:51.531211 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:51.531213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:51.531378 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:51.531468 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:51.531473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.532232 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.532235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:51.532791 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:51.532800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.533182 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.533184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:51.533242 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:51.533679 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:51.533690 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:51.535294 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
W20260926 14:50:51.535545 21751 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:51.530809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:42613 (local address 127.12.68.254:36535)
0926 14:50:51.531025 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:51.531029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:51.531041 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:51.531075 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:51.531077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:51.531116 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:51.531188 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:51.531584 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.532141 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.532885 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.533105 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.533175 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:51.535432 (+ 2257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:42613: BlockingRecv error: recv got EOF from 127.12.68.193:42613 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
Sep 26 14:50:51 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:51 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:51.699678 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:51.694691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:33041)
0926 14:50:51.694965 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:51.694980 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:51.695089 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:51.695351 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:51.695355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:51.695556 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:51.695674 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:51.695683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.696647 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.696650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:51.697199 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:51.697207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.697559 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.697561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:51.697624 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:51.698021 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:51.698031 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:51.699559 (+ 1528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":69,"threads_started":1}
W20260926 14:50:51.699777 21751 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:51.694768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:33041 (local address 127.12.68.254:36535)
0926 14:50:51.694929 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:51.694934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:51.695050 (+ 116us) server_negotiation.cc:407] Connection header received
0926 14:50:51.695181 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:51.695184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:51.695234 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:51.695306 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:51.695780 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.696558 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.697287 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.697491 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.697553 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:51.699683 (+ 2130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:33041: BlockingRecv error: recv got EOF from 127.12.68.194:33041 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 14:50:51 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:51 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:51.883036 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:51.878323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:57959)
0926 14:50:51.878570 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:51.878583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:51.878674 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:51.878873 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:51.878875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:51.879050 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:51.879149 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:51.879155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.879831 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.879833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:51.880418 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:51.880427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.880780 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.880782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:51.880840 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:51.881256 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:51.881267 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:51.882924 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
W20260926 14:50:51.883158 21751 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:51.878420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:57959 (local address 127.12.68.254:36535)
0926 14:50:51.878542 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:51.878545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:51.878630 (+ 85us) server_negotiation.cc:407] Connection header received
0926 14:50:51.878748 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:51.878750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:51.878792 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:51.878856 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:51.879234 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.879755 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.880498 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.880706 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.880773 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:51.883044 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:57959: BlockingRecv error: recv got EOF from 127.12.68.195:57959 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 14:50:52 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:52 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:52.544863 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:52.536069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:47775)
0926 14:50:52.536344 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:52.536356 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:52.536454 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:52.536668 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:52.536669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:52.536853 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:52.536945 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:52.536950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.537635 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.537638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:52.538184 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:52.538190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.538571 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.538573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:52.538628 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:52.543073 (+ 4445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:52.543085 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:52.544755 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
W20260926 14:50:52.545013 21755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:52.536127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:47775 (local address 127.12.68.254:36535)
0926 14:50:52.536341 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:52.536346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:52.536418 (+ 72us) server_negotiation.cc:407] Connection header received
0926 14:50:52.536536 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:52.536539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:52.536582 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:52.536646 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:52.537034 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.537556 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.538267 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.538493 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.538566 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:52.544876 (+ 6310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:47775: BlockingRecv error: recv got EOF from 127.12.68.193:47775 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":52,"threads_started":1}
Sep 26 14:50:52 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:52 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:52.705498 21756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:52.700322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:40809)
0926 14:50:52.700555 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:52.700570 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:52.700668 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:52.700870 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:52.700872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:52.701062 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:52.701152 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:52.701157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.701866 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.701869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:52.702573 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:52.702583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.703109 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.703112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:52.703187 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:52.703649 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:52.703660 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:52.705374 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":71,"threads_started":1}
W20260926 14:50:52.705605 21755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:52.700399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:40809 (local address 127.12.68.254:36535)
0926 14:50:52.700539 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:52.700542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:52.700632 (+ 90us) server_negotiation.cc:407] Connection header received
0926 14:50:52.700739 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:52.700741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:52.700783 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:52.700848 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:52.701249 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.701762 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.702682 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.702993 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.703078 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:52.705491 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:40809: BlockingRecv error: recv got EOF from 127.12.68.194:40809 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 14:50:52 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:52 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:52.888507 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:52.883728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:48317)
0926 14:50:52.883984 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:52.883999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:52.884118 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:52.884360 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:52.884362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:52.884529 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:52.884623 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:52.884628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.885310 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.885312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:52.885856 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:52.885862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.886220 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.886223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:52.886283 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:52.886705 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:52.886715 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:52.888362 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":68,"threads_started":1}
W20260926 14:50:52.888651 21755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:52.883779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:48317 (local address 127.12.68.254:36535)
0926 14:50:52.883914 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:52.883919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:52.884067 (+ 148us) server_negotiation.cc:407] Connection header received
0926 14:50:52.884212 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:52.884215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:52.884268 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:52.884330 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:52.884717 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.885236 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.885932 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.886141 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.886207 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:52.888523 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:48317: BlockingRecv error: recv got EOF from 127.12.68.195:48317 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 14:50:53 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:53 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:53.551018 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:53.545699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:60245)
0926 14:50:53.546012 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:53.546025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:53.546103 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:53.546367 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:53.546369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:53.546549 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:53.546649 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:53.546657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.547452 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.547455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:53.548045 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:53.548054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.548485 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.548489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:53.548571 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:53.549059 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:53.549073 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:53.550880 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":90,"threads_started":1}
W20260926 14:50:53.551141 21759 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:53.545839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:60245 (local address 127.12.68.254:36535)
0926 14:50:53.546047 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:53.546050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:53.546083 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:50:53.546213 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:53.546215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:53.546257 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:53.546324 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:53.546737 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.547353 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.548154 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.548374 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.548471 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:53.551015 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:60245: BlockingRecv error: recv got EOF from 127.12.68.193:60245 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":51,"threads_started":1}
Sep 26 14:50:53 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:53 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:53.717218 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:53.706191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:47687)
0926 14:50:53.706447 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:53.706462 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:53.706576 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:53.706809 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:53.706812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:53.707001 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:53.707092 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:53.707097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.707840 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.707844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:53.708450 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:53.708459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.708880 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.708883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:53.708958 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:53.715185 (+ 6227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:53.715197 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:53.717103 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
W20260926 14:50:53.717320 21759 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:53.706274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:47687 (local address 127.12.68.254:36535)
0926 14:50:53.706434 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:53.706439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:53.706537 (+ 98us) server_negotiation.cc:407] Connection header received
0926 14:50:53.706661 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:53.706664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:53.706710 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:53.706772 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:53.707199 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.707737 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.708571 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.708785 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.708869 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:53.717214 (+ 8345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:47687: BlockingRecv error: recv got EOF from 127.12.68.194:47687 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 14:50:53 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:53 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:53.894040 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:53.889158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:57301)
0926 14:50:53.889390 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:53.889404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:53.889502 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:53.889720 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:53.889723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:53.889903 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:53.889988 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:53.889993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.890721 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.890724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:53.891294 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:53.891302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.891735 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.891739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:53.891818 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:53.892287 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:53.892297 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:53.893894 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":70,"threads_started":1}
W20260926 14:50:53.894117 21759 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:53.889264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:57301 (local address 127.12.68.254:36535)
0926 14:50:53.889392 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:53.889396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:53.889464 (+ 68us) server_negotiation.cc:407] Connection header received
0926 14:50:53.889581 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:53.889584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:53.889630 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:53.889690 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:53.890103 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.890624 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.891395 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.891637 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.891722 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:53.894019 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:57301: BlockingRecv error: recv got EOF from 127.12.68.195:57301 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 14:50:54 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:54 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:54.556784 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.551647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:43287)
0926 14:50:54.551911 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:54.551923 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:54.552023 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:54.552264 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:54.552266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:54.552438 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:54.552538 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:54.552544 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.553330 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.553333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:54.553893 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:54.553902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.554286 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.554288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:54.554351 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:54.554775 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:54.554787 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:54.556601 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
W20260926 14:50:54.556841 21763 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:54.551732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:43287 (local address 127.12.68.254:36535)
0926 14:50:54.551979 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:54.551983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:54.551995 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:54.552098 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:54.552100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:54.552148 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:54.552216 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:54.552621 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.553217 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.553986 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.554203 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.554269 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:54.556730 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:43287: BlockingRecv error: recv got EOF from 127.12.68.193:43287 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":53,"threads_started":1}
Sep 26 14:50:54 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:54 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:54.722995 21764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.717897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:50603)
0926 14:50:54.718149 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:54.718162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:54.718261 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:54.718461 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:54.718463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:54.718629 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:54.718720 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:54.718725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.719411 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.719415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:54.720222 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:54.720233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.720741 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.720745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:54.720828 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:54.721256 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:54.721266 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:54.722839 (+ 1573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
W20260926 14:50:54.723073 21763 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:54.718009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:50603 (local address 127.12.68.254:36535)
0926 14:50:54.718154 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:54.718157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:54.718223 (+ 66us) server_negotiation.cc:407] Connection header received
0926 14:50:54.718331 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:54.718334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:54.718376 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:54.718437 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:54.718800 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.719302 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.720338 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.720631 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.720714 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:54.722977 (+ 2263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:50603: BlockingRecv error: recv got EOF from 127.12.68.194:50603 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 14:50:54 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:54 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:54.899413 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.894720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:33719)
0926 14:50:54.894970 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:54.894982 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:54.895077 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:54.895290 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:54.895292 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:54.895456 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:54.895547 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:54.895552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.896263 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.896266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:54.896827 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:54.896835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.897223 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.897226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:54.897298 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:54.897725 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:54.897735 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:54.899284 (+ 1549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
W20260926 14:50:54.899509 21763 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:54.894777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:33719 (local address 127.12.68.254:36535)
0926 14:50:54.894887 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:54.894891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:54.895040 (+ 149us) server_negotiation.cc:407] Connection header received
0926 14:50:54.895155 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:54.895158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:54.895201 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:54.895264 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:54.895633 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.896153 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.896920 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.897125 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.897186 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:54.899409 (+ 2223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:33719: BlockingRecv error: recv got EOF from 127.12.68.195:33719 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 14:50:55 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:55 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:55.562248 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:55.557404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:48127)
0926 14:50:55.557662 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:55.557675 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:55.557762 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:55.557971 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:55.557973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:55.558137 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:55.558228 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:55.558233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:55.558960 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:55.558963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:55.559497 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:55.559503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:55.559949 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:55.559953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:55.560016 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:55.560486 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:55.560497 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:55.562135 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":61,"threads_started":1}
W20260926 14:50:55.562422 21767 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:55.557474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:48127 (local address 127.12.68.254:36535)
0926 14:50:55.557707 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:55.557711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:55.557722 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:50:55.557835 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:55.557837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:55.557877 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:55.557944 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:55.558361 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:55.558882 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:55.559595 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:55.559819 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:55.559916 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:55.562296 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:48127: BlockingRecv error: recv got EOF from 127.12.68.193:48127 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
Sep 26 14:50:55 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:55 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:55.728878 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:55.723675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:38059)
0926 14:50:55.723965 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:55.723978 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:55.724116 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:55.724343 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:55.724346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:55.724543 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:55.724634 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:55.724640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:55.725394 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:55.725398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:55.725947 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:55.725955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:55.726372 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:55.726375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:55.726451 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:55.726895 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:55.726905 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:55.728764 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":70,"threads_started":1}
W20260926 14:50:55.728999 21767 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:55.723754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:38059 (local address 127.12.68.254:36535)
0926 14:50:55.723873 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:55.723908 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:55.724062 (+ 154us) server_negotiation.cc:407] Connection header received
0926 14:50:55.724197 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:55.724200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:55.724245 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:55.724310 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:55.724744 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:55.725295 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:55.726064 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:55.726278 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:55.726360 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:55.728889 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:38059: BlockingRecv error: recv got EOF from 127.12.68.194:38059 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 14:50:55 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:55 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:55.904985 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:55.900098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:36139)
0926 14:50:55.900356 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:55.900368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:55.900473 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:55.900706 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:55.900709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:55.900909 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:55.900998 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:55.901003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:55.901718 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:55.901722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:55.902268 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:55.902275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:55.902676 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:55.902679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:55.902752 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:55.903189 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:55.903199 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:55.904842 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
W20260926 14:50:55.905082 21767 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:55.900164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:36139 (local address 127.12.68.254:36535)
0926 14:50:55.900284 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:55.900289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:55.900433 (+ 144us) server_negotiation.cc:407] Connection header received
0926 14:50:55.900558 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:55.900561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:55.900604 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:55.900674 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:55.901106 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:55.901621 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:55.902373 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:55.902584 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:55.902665 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:55.904973 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:36139: BlockingRecv error: recv got EOF from 127.12.68.195:36139 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 14:50:56 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:56 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:56.567854 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.562969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:34629)
0926 14:50:56.563225 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:56.563240 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:56.563345 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:56.563572 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:56.563574 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:56.563760 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:56.563854 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.563859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.564608 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.564611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.565168 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.565175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.565553 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.565555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.565610 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.566030 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:56.566040 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:56.567740 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
W20260926 14:50:56.568114 21771 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:56.563027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:34629 (local address 127.12.68.254:36535)
0926 14:50:56.563307 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:56.563311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:56.563327 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:50:56.563431 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:56.563433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:56.563477 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:56.563545 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:56.563963 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.564510 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.565258 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.565477 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.565546 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.567857 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:34629: BlockingRecv error: recv got EOF from 127.12.68.193:34629 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":54,"threads_started":1}
Sep 26 14:50:56 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:56 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:56.734797 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.729571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:39861)
0926 14:50:56.729835 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:56.729851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:56.729953 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:56.730191 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:56.730194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:56.730385 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:56.730478 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.730484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.731263 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.731267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.731997 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.732008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.732468 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.732471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.732540 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.733015 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:56.733027 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:56.734651 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
W20260926 14:50:56.734885 21771 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:56.729751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:39861 (local address 127.12.68.254:36535)
0926 14:50:56.729885 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:56.729890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:56.729903 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:56.730039 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:56.730042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:56.730095 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:56.730160 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:56.730590 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.731167 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.732113 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.732349 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.732433 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.734787 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:39861: BlockingRecv error: recv got EOF from 127.12.68.194:39861 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 14:50:56 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:56 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:56.910696 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.905672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:51111)
0926 14:50:56.905961 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:56.905973 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:56.906077 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:56.906317 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:56.906320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:56.906511 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:56.906600 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.906605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.907321 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.907325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.907865 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.907872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.908322 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.908325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.908385 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.908873 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:56.908885 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:56.910556 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":97,"threads_started":1}
W20260926 14:50:56.910797 21771 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:56.905774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:51111 (local address 127.12.68.254:36535)
0926 14:50:56.905911 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:56.905916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:56.906036 (+ 120us) server_negotiation.cc:407] Connection header received
0926 14:50:56.906170 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:56.906173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:56.906221 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:56.906282 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:56.906709 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.907227 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.908010 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.908238 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.908313 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.910684 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:51111: BlockingRecv error: recv got EOF from 127.12.68.195:51111 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 14:50:57 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:57 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:57.576336 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:57.568605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:38805)
0926 14:50:57.568882 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:57.568896 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:57.568986 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:57.569281 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:57.569284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:57.569468 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:57.569582 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:57.569588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:57.570300 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:57.570303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:57.570851 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:57.570858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:57.571228 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:57.571230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:57.571286 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:57.574442 (+ 3156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:57.574457 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:57.576224 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
W20260926 14:50:57.576469 21775 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:57.568637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:38805 (local address 127.12.68.254:36535)
0926 14:50:57.568951 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:57.568954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:57.568966 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:57.569076 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:57.569079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:57.569128 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:57.569206 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:57.569676 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:57.570214 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:57.570935 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:57.571153 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:57.571218 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:57.576331 (+ 5113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:38805: BlockingRecv error: recv got EOF from 127.12.68.193:38805 (error 108)
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":81,"threads_started":1}
Sep 26 14:50:57 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:57 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:57.740634 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:57.735589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:55499)
0926 14:50:57.735851 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:57.735864 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:57.735984 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:57.736225 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:57.736228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:57.736420 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:57.736505 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:57.736511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:57.737246 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:57.737249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:57.737807 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:57.737815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:57.738229 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:57.738232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:57.738308 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:57.738748 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:57.738758 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:57.740519 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
W20260926 14:50:57.740811 21775 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:57.735614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:55499 (local address 127.12.68.254:36535)
0926 14:50:57.735778 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:57.735782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:57.735941 (+ 159us) server_negotiation.cc:407] Connection header received
0926 14:50:57.736070 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:57.736073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:57.736118 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:57.736187 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:57.736606 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:57.737148 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:57.737910 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:57.738133 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:57.738215 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:57.740696 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:55499: BlockingRecv error: recv got EOF from 127.12.68.194:55499 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 14:50:57 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:57 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.195@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:57.916426 21777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:57.911417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.195:37225)
0926 14:50:57.911671 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:57.911684 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:57.911788 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:57.912065 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:57.912068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:57.912295 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:57.912397 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:57.912404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:57.913122 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:57.913124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:57.913689 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:57.913696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:57.914109 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:57.914111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:57.914168 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:57.914580 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:57.914590 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:57.916272 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":99,"threads_started":1}
W20260926 14:50:57.916528 21775 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:57.911478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:37225 (local address 127.12.68.254:36535)
0926 14:50:57.911604 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:57.911609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:57.911760 (+ 151us) server_negotiation.cc:407] Connection header received
0926 14:50:57.911874 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:57.911896 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:57.911949 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:57.912018 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:57.912524 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:57.913039 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:57.913803 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:57.914026 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:57.914102 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:57.916400 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.195:37225: BlockingRecv error: recv got EOF from 127.12.68.195:37225 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 14:50:58 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:58 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.193@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:58.582214 21778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:58.577234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.193:44481)
0926 14:50:58.577486 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:58.577499 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:58.577581 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:58.577790 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:58.577792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:58.577965 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:58.578054 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:58.578059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:58.578801 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:58.578804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:58.579357 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:58.579365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:58.579765 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:58.579767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:58.579823 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:58.580272 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:58.580283 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:58.582104 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
W20260926 14:50:58.582379 21779 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:58.577341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:44481 (local address 127.12.68.254:36535)
0926 14:50:58.577536 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:58.577540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:58.577552 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:58.577658 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:58.577660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:58.577700 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:58.577767 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:58.578144 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:58.578699 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:58.579449 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:58.579683 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:58.579755 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:58.582235 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.193:44481: BlockingRecv error: recv got EOF from 127.12.68.193:44481 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":51,"threads_started":1}
Sep 26 14:50:58 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:58 dist-test-slave-24mn krb5kdc[20721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.68.194@KRBTEST.COM for oryx/127.12.68.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:58.746413 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:58.741480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36535 (local address 127.12.68.194:50521)
0926 14:50:58.741749 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:58.741762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:58.741859 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:58.742085 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:58.742087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:58.742273 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:58.742362 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:58.742367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:58.743077 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:58.743080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:58.743644 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:58.743652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:58.744132 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:58.744135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:58.744212 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:58.744668 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:58.744679 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:58.746249 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.68.254:36535: Server oryx/127.12.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
W20260926 14:50:58.746475 21779 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:58.741565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:50521 (local address 127.12.68.254:36535)
0926 14:50:58.741706 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:58.741710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:58.741820 (+ 110us) server_negotiation.cc:407] Connection header received
0926 14:50:58.741939 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:58.741942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:58.741988 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:58.742051 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:58.742462 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:58.742982 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:58.743747 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:58.744036 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:58.744129 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:58.746379 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.68.194:50521: BlockingRecv error: recv got EOF from 127.12.68.194:50521 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:50:58.818863 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 21304
I20260926 14:50:58.823698 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 21456
I20260926 14:50:58.827848 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 21588
I20260926 14:50:58.831691 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 21234
2026-09-26T14:50:58Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17683 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T14:50:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:58Z Disabled control of system clock
I20260926 14:50:58.858654 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:38057
--webserver_interface=127.12.68.254
--webserver_port=0
--builtin_ntp_servers=127.12.68.212:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:38057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 14:50:58.960464 21784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:58.960697 21784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:58.960742 21784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:58.964653 21784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:58.964699 21784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:58.964721 21784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:58.964743 21784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:58.964771 21784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:58.969616 21784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:38057
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:38057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:58.970541 21784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:58.971784 21784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:50:58.979983 21784 server_base.cc:1034] running on GCE node
W20260926 14:50:58.979940 21790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:58.979943 21792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:58.979940 21789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:58.980713 21784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:58.981685 21784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:58.982837 21784 hybrid_clock.cc:648] HybridClock initialized: now 1790434258982815 us; error 39 us; skew 500 ppm
I20260926 14:50:58.984472 21784 webserver.cc:460] Webserver started at http://127.12.68.254:41279/ using document root <none> and password file <none>
I20260926 14:50:58.984962 21784 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:58.985018 21784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:58.985219 21784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:58.987185 21784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "5b5ec92fb67141ab9a642eb4afd29e4d"
format_stamp: "Formatted at 2026-09-26 14:50:58 on dist-test-slave-24mn"
server_key: "a6936678a0a8d9eaa550db9a165b3338"
server_key_iv: "f95bca2e9805ca259945f3b3f0bbc04a"
server_key_version: "encryptionkey@0"
I20260926 14:50:58.987670 21784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "5b5ec92fb67141ab9a642eb4afd29e4d"
format_stamp: "Formatted at 2026-09-26 14:50:58 on dist-test-slave-24mn"
server_key: "a6936678a0a8d9eaa550db9a165b3338"
server_key_iv: "f95bca2e9805ca259945f3b3f0bbc04a"
server_key_version: "encryptionkey@0"
I20260926 14:50:58.990948 21784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:50:58.993076 21798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:58.994177 21784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:58.994280 21784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "5b5ec92fb67141ab9a642eb4afd29e4d"
format_stamp: "Formatted at 2026-09-26 14:50:58 on dist-test-slave-24mn"
server_key: "a6936678a0a8d9eaa550db9a165b3338"
server_key_iv: "f95bca2e9805ca259945f3b3f0bbc04a"
server_key_version: "encryptionkey@0"
I20260926 14:50:58.994372 21784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:59.014437 21784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:59.015045 21784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:59.015213 21784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:59.021338 21784 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:38057
I20260926 14:50:59.021351 21850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:38057 every 8 connection(s)
I20260926 14:50:59.022192 21784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:59.023289 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 21784
I20260926 14:50:59.023396 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:59.023684 12563 external_mini_cluster.cc:1428] Setting key 8cb94c528a82f3c08f7af1b03c711912
I20260926 14:50:59.025642 21851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:59.030740 21851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d: Bootstrap starting.
I20260926 14:50:59.032275 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:59.025061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.12.68.254:38057)
0926 14:50:59.025701 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:59.025715 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:59.025755 (+ 40us) server_negotiation.cc:407] Connection header received
0926 14:50:59.026386 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:59.026402 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:59.026778 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:59.027066 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:59.027447 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:59.028433 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:59.029296 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:59.029742 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:59.029876 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:59.030681 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:59.030686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:59.030697 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:59.031067 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:59.031121 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:59.031141 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:59.031226 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:59.031355 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 14:50:59.031582 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":139,"threads_started":1}
I20260926 14:50:59.032783 21851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:59.033425 21851 log.cc:824] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:59.035043 21851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d: No bootstrap required, opened a new log
I20260926 14:50:59.038110 21851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ec92fb67141ab9a642eb4afd29e4d" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38057 } }
I20260926 14:50:59.038300 21851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:59.038359 21851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5ec92fb67141ab9a642eb4afd29e4d, State: Initialized, Role: FOLLOWER
I20260926 14:50:59.038740 21851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ec92fb67141ab9a642eb4afd29e4d" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38057 } }
I20260926 14:50:59.038861 21851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:59.038921 21851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:59.039009 21851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:59.039732 21851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ec92fb67141ab9a642eb4afd29e4d" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38057 } }
I20260926 14:50:59.040037 21851 leader_election.cc:302] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b5ec92fb67141ab9a642eb4afd29e4d; no voters:
I20260926 14:50:59.040280 21851 leader_election.cc:288] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:59.040359 21856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:59.040563 21856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [term 1 LEADER]: Becoming Leader. State: Replica: 5b5ec92fb67141ab9a642eb4afd29e4d, State: Running, Role: LEADER
I20260926 14:50:59.040869 21856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ec92fb67141ab9a642eb4afd29e4d" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38057 } }
I20260926 14:50:59.041306 21851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:59.042284 21858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b5ec92fb67141ab9a642eb4afd29e4d. Latest consensus state: current_term: 1 leader_uuid: "5b5ec92fb67141ab9a642eb4afd29e4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ec92fb67141ab9a642eb4afd29e4d" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38057 } } }
I20260926 14:50:59.042400 21858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:59.042757 21865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:59.042762 21857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b5ec92fb67141ab9a642eb4afd29e4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ec92fb67141ab9a642eb4afd29e4d" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38057 } } }
I20260926 14:50:59.042889 21857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:59.045732 21865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:59.050686 21865 catalog_manager.cc:1347] Generated new cluster ID: ee5048ec8d8345c9a665bb1f4e557b73
I20260926 14:50:59.050773 21865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:59.054975 21865 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:59.056137 21865 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:59.067667 21865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b5ec92fb67141ab9a642eb4afd29e4d: Generated new TSK 0
I20260926 14:50:59.068388 21865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:50:59.072566 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:38057
--builtin_ntp_servers=127.12.68.212:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 14:50:59.175261 21875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:59.175488 21875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:59.175541 21875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:59.179436 21875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:59.179482 21875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:59.179549 21875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:59.184156 21875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:38057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:59.185074 21875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:59.186335 21875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:59.195015 21880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:59.195034 21883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:59.195007 21881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:59.195047 21875 server_base.cc:1034] running on GCE node
I20260926 14:50:59.195575 21875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:59.196192 21875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:59.197355 21875 hybrid_clock.cc:648] HybridClock initialized: now 1790434259197336 us; error 29 us; skew 500 ppm
I20260926 14:50:59.199049 21875 webserver.cc:460] Webserver started at http://127.12.68.193:43169/ using document root <none> and password file <none>
I20260926 14:50:59.199558 21875 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:59.199621 21875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:59.199792 21875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:59.201733 21875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "887ec0f05dfb4e53adaa7821eb60ee9e"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "3bd2a5441c2cc76e885cc902b53e8dd2"
server_key_iv: "85f1c1185896156dee94b42946b172ce"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.202242 21875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "887ec0f05dfb4e53adaa7821eb60ee9e"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "3bd2a5441c2cc76e885cc902b53e8dd2"
server_key_iv: "85f1c1185896156dee94b42946b172ce"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.205569 21875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:50:59.207784 21889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:59.208890 21875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:59.208981 21875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "887ec0f05dfb4e53adaa7821eb60ee9e"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "3bd2a5441c2cc76e885cc902b53e8dd2"
server_key_iv: "85f1c1185896156dee94b42946b172ce"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.209075 21875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:59.220075 21875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:59.220623 21875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:59.220777 21875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:59.221275 21875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:59.222108 21875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:59.222151 21875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:59.222191 21875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:59.222220 21875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:59.230175 21875 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:34571
I20260926 14:50:59.230221 22002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:34571 every 8 connection(s)
I20260926 14:50:59.231009 21875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:59.237802 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 21875
I20260926 14:50:59.237907 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:59.238159 12563 external_mini_cluster.cc:1428] Setting key 11f88f6e3606ed44a276e3289f14a7f8
I20260926 14:50:59.239498 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:59.232651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:57855 (local address 127.12.68.254:38057)
0926 14:50:59.232766 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:59.232769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:59.233758 (+ 989us) server_negotiation.cc:407] Connection header received
0926 14:50:59.235067 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:59.235071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:59.235131 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:59.235203 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:59.236086 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:59.236608 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:59.237693 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:59.237903 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:59.237988 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:59.239041 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:59.239045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:59.239047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:59.239059 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:59.239078 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:59.239086 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:59.239147 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:59.239287 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 14:50:59.239362 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:50:59.239961 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:38057
--builtin_ntp_servers=127.12.68.212:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 14:50:59.240315 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:59.233013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:38057 (local address 127.12.68.193:57855)
0926 14:50:59.233615 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:59.233653 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:59.234871 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:59.235378 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:59.235384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:59.235464 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:59.235960 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:59.235974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:59.236726 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:59.236729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:59.237578 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:59.237586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:59.238070 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:59.238074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:59.238217 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:59.238787 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:59.238788 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:59.238826 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:59.238887 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:59.239159 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:59.239168 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:59.239174 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:50:59.239276 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:50:59.239537 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":107,"threads_started":1}
I20260926 14:50:59.241504 22003 heartbeater.cc:347] Connected to a master server at 127.12.68.254:38057
I20260926 14:50:59.241703 22003 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:59.242188 22003 heartbeater.cc:511] Master 127.12.68.254:38057 requested a full tablet report, sending...
I20260926 14:50:59.243661 21815 ts_manager.cc:194] Registered new tserver with Master: 887ec0f05dfb4e53adaa7821eb60ee9e (127.12.68.193:34571)
I20260926 14:50:59.245185 21815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.68.193:57855
W20260926 14:50:59.344112 22006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:59.344324 22006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:59.344357 22006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:59.348198 22006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:59.348241 22006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:59.348289 22006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:50:59.352712 22006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:38057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:59.353567 22006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:59.354745 22006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:59.363318 22011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:59.363338 22012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:59.363396 22006 server_base.cc:1034] running on GCE node
W20260926 14:50:59.363318 22014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:59.364060 22006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:59.364665 22006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:59.365830 22006 hybrid_clock.cc:648] HybridClock initialized: now 1790434259365809 us; error 33 us; skew 500 ppm
I20260926 14:50:59.367519 22006 webserver.cc:460] Webserver started at http://127.12.68.194:34195/ using document root <none> and password file <none>
I20260926 14:50:59.368064 22006 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:59.368129 22006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:59.368330 22006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:59.371632 22006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "655e21ffdefa473cac9407c73d9c697d"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "b1113f7e6002281f39c8978485f45958"
server_key_iv: "337cfe6a79712d98a8981a6f6a8ce059"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.372164 22006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "655e21ffdefa473cac9407c73d9c697d"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "b1113f7e6002281f39c8978485f45958"
server_key_iv: "337cfe6a79712d98a8981a6f6a8ce059"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.375463 22006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:59.377841 22020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:59.378937 22006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 14:50:59.379034 22006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "655e21ffdefa473cac9407c73d9c697d"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "b1113f7e6002281f39c8978485f45958"
server_key_iv: "337cfe6a79712d98a8981a6f6a8ce059"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.379124 22006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:59.394366 22006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:59.394948 22006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:59.395092 22006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:59.395596 22006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:59.396500 22006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:59.396545 22006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:59.396595 22006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:59.396620 22006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:59.404182 22006 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:46553
I20260926 14:50:59.404214 22133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:46553 every 8 connection(s)
I20260926 14:50:59.404955 22006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:59.406241 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 22006
I20260926 14:50:59.406337 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:59.406648 12563 external_mini_cluster.cc:1428] Setting key 9b3b15544a28023513e2bdaeafde7372
I20260926 14:50:59.408430 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:38057
--builtin_ntp_servers=127.12.68.212:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 14:50:59.413789 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:59.406601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:57467 (local address 127.12.68.254:38057)
0926 14:50:59.406744 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:59.406749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:59.407806 (+ 1057us) server_negotiation.cc:407] Connection header received
0926 14:50:59.409238 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:59.409241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:59.409293 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:59.409376 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:59.410406 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:59.410918 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:59.412065 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:59.412300 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:59.412385 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:59.413373 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:59.413376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:59.413379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:59.413390 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:59.413410 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:59.413418 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:59.413472 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:59.413597 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 14:50:59.413655 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:50:59.414529 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:59.406990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:38057 (local address 127.12.68.194:57467)
0926 14:50:59.407671 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:59.407718 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:59.409041 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:59.409532 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:59.409541 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:59.409658 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:59.410266 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:59.410282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:59.411018 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:59.411021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:59.411964 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:59.411973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:59.412423 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:59.412481 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:59.412640 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:59.413146 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:59.413148 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:59.413185 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:59.413241 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:59.413481 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:59.413489 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:59.413494 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:59.413590 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 14:50:59.413847 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":127,"threads_started":1}
I20260926 14:50:59.415623 22134 heartbeater.cc:347] Connected to a master server at 127.12.68.254:38057
I20260926 14:50:59.415858 22134 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:59.416447 22134 heartbeater.cc:511] Master 127.12.68.254:38057 requested a full tablet report, sending...
I20260926 14:50:59.417461 21815 ts_manager.cc:194] Registered new tserver with Master: 655e21ffdefa473cac9407c73d9c697d (127.12.68.194:46553)
I20260926 14:50:59.417989 21815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.68.194:57467
W20260926 14:50:59.513186 22137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:59.513435 22137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:59.513480 22137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:59.517375 22137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:59.517417 22137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:59.517468 22137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:50:59.522710 22137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:38057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:59.523638 22137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:59.524909 22137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:59.533501 22143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:59.533481 22145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:59.533497 22142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:59.533746 22137 server_base.cc:1034] running on GCE node
I20260926 14:50:59.534080 22137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:59.534626 22137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:59.535776 22137 hybrid_clock.cc:648] HybridClock initialized: now 1790434259535758 us; error 37 us; skew 500 ppm
I20260926 14:50:59.537410 22137 webserver.cc:460] Webserver started at http://127.12.68.195:37933/ using document root <none> and password file <none>
I20260926 14:50:59.537902 22137 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:59.537963 22137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:59.538126 22137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:59.539973 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "a6bac43b83bb459d8ab971bc4e1cad50"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "9118090b9a7ba67cba54b028fd57790b"
server_key_iv: "627792c6eb2c4e72f5a8ab4b97d394a4"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.540463 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "a6bac43b83bb459d8ab971bc4e1cad50"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "9118090b9a7ba67cba54b028fd57790b"
server_key_iv: "627792c6eb2c4e72f5a8ab4b97d394a4"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.543946 22137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:59.546132 22151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:59.547256 22137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:59.547341 22137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "a6bac43b83bb459d8ab971bc4e1cad50"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "9118090b9a7ba67cba54b028fd57790b"
server_key_iv: "627792c6eb2c4e72f5a8ab4b97d394a4"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.547435 22137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:59.561439 22137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:59.562043 22137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:59.562180 22137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:59.562673 22137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:59.563488 22137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:59.563530 22137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:59.563567 22137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:59.563596 22137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:59.571265 22137 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:34447
I20260926 14:50:59.571311 22264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:34447 every 8 connection(s)
I20260926 14:50:59.572072 22137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:59.574225 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 22137
I20260926 14:50:59.574314 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:59.574546 12563 external_mini_cluster.cc:1428] Setting key bb322321b0518c56907e9a02d77d5321
I20260926 14:50:59.580300 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:59.573557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:52353 (local address 127.12.68.254:38057)
0926 14:50:59.573685 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:59.573690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:59.574748 (+ 1058us) server_negotiation.cc:407] Connection header received
0926 14:50:59.576185 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:59.576188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:59.576239 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:59.576307 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:59.577121 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:59.577621 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:59.578676 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:59.578882 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:59.578963 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:59.579862 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:59.579867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:59.579869 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:59.579901 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:59.579922 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:59.579931 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:59.579990 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:59.580116 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 14:50:59.580171 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:50:59.581036 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:59.573948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:38057 (local address 127.12.68.195:52353)
0926 14:50:59.574595 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:59.574641 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:59.576011 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:59.576487 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:59.576493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:59.576572 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:59.577001 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:59.577012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:59.577732 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:59.577735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:59.578563 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:59.578571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:59.579001 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:59.579005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:59.579143 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:59.579638 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:59.579640 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:59.579677 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:59.579732 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:59.579999 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:59.580007 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:59.580013 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:50:59.580109 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 14:50:59.580386 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":99,"threads_started":1}
I20260926 14:50:59.581981 22265 heartbeater.cc:347] Connected to a master server at 127.12.68.254:38057
I20260926 14:50:59.582176 22265 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:59.582690 22265 heartbeater.cc:511] Master 127.12.68.254:38057 requested a full tablet report, sending...
I20260926 14:50:59.583602 21815 ts_manager.cc:194] Registered new tserver with Master: a6bac43b83bb459d8ab971bc4e1cad50 (127.12.68.195:34447)
I20260926 14:50:59.584182 21815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.68.195:52353
I20260926 14:50:59.587698 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 14:50:59.590104 21854 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:59.589392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.12.68.254:38057)
0926 14:50:59.589514 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:59.589519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:59.589649 (+ 130us) server_negotiation.cc:407] Connection header received
0926 14:50:59.589743 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:59.589745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:59.589778 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:59.589829 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:59.590004 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59198: BlockingRecv error: recv got EOF from 127.0.0.1:59198 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:50:59.590466 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 21875
I20260926 14:50:59.595374 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 22006
I20260926 14:50:59.599746 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 22137
I20260926 14:50:59.603829 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 21784
2026-09-26T14:50:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (771 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T14:50:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:59Z Disabled control of system clock
I20260926 14:50:59.629611 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:43577
--webserver_interface=127.12.68.254
--webserver_port=0
--builtin_ntp_servers=127.12.68.212:35431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:43577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 14:50:59.731582 22276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:59.731805 22276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:59.731838 22276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:59.735737 22276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:59.735785 22276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:59.735798 22276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:59.735813 22276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:59.735826 22276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:59.740350 22276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:35431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:43577
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:43577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:59.741261 22276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:59.742496 22276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:59.749969 22281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:59.750099 22276 server_base.cc:1034] running on GCE node
W20260926 14:50:59.749979 22282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:59.749986 22284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:59.750746 22276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:59.751689 22276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:59.752835 22276 hybrid_clock.cc:648] HybridClock initialized: now 1790434259752816 us; error 29 us; skew 500 ppm
I20260926 14:50:59.754704 22276 webserver.cc:460] Webserver started at http://127.12.68.254:37225/ using document root <none> and password file <none>
I20260926 14:50:59.755206 22276 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:59.755263 22276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:59.755432 22276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:59.757337 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "c2f8d76faa67494ba8ca28c99feffb7c"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "ae128a11319ee5de78bafa995f95977d"
server_key_iv: "b2c9a79e2cd94ef0500a5f137c00815a"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.757839 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "c2f8d76faa67494ba8ca28c99feffb7c"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "ae128a11319ee5de78bafa995f95977d"
server_key_iv: "b2c9a79e2cd94ef0500a5f137c00815a"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.761159 22276 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:50:59.763410 22290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:59.764549 22276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:59.764667 22276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "c2f8d76faa67494ba8ca28c99feffb7c"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "ae128a11319ee5de78bafa995f95977d"
server_key_iv: "b2c9a79e2cd94ef0500a5f137c00815a"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.764753 22276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:59.768913 22276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:59.769434 22276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:59.769575 22276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:59.775462 22276 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:43577
I20260926 14:50:59.775496 22342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:43577 every 8 connection(s)
I20260926 14:50:59.776298 22276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:59.779191 22343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:59.784078 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 22276
I20260926 14:50:59.784190 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:50:59.784273 22343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c: Bootstrap starting.
I20260926 14:50:59.784437 12563 external_mini_cluster.cc:1428] Setting key 8438a03b1bb4cff45290d0b375bfbd57
I20260926 14:50:59.786662 22343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:59.787484 22343 log.cc:824] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:59.789510 22343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c: No bootstrap required, opened a new log
I20260926 14:50:59.789856 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:59.785646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60624 (local address 127.12.68.254:43577)
0926 14:50:59.786266 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:59.786284 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:59.786323 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:50:59.787100 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:59.787121 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:59.787540 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:59.787926 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:59.788666 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:59.788670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:59.788681 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:59.788690 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:59.788755 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:59.788771 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:59.788849 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:59.788964 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 14:50:59.789191 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
I20260926 14:50:59.793123 22343 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f8d76faa67494ba8ca28c99feffb7c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43577 } }
I20260926 14:50:59.793319 22343 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:59.793380 22343 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f8d76faa67494ba8ca28c99feffb7c, State: Initialized, Role: FOLLOWER
I20260926 14:50:59.793758 22343 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f8d76faa67494ba8ca28c99feffb7c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43577 } }
I20260926 14:50:59.793879 22343 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:59.793937 22343 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:59.794028 22343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:59.794811 22343 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f8d76faa67494ba8ca28c99feffb7c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43577 } }
I20260926 14:50:59.795110 22343 leader_election.cc:302] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2f8d76faa67494ba8ca28c99feffb7c; no voters:
I20260926 14:50:59.795380 22343 leader_election.cc:288] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:59.795449 22348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:59.795644 22348 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [term 1 LEADER]: Becoming Leader. State: Replica: c2f8d76faa67494ba8ca28c99feffb7c, State: Running, Role: LEADER
I20260926 14:50:59.795945 22348 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f8d76faa67494ba8ca28c99feffb7c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43577 } }
I20260926 14:50:59.796484 22343 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:59.797333 22349 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2f8d76faa67494ba8ca28c99feffb7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f8d76faa67494ba8ca28c99feffb7c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43577 } } }
I20260926 14:50:59.797456 22349 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:59.797691 22350 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2f8d76faa67494ba8ca28c99feffb7c. Latest consensus state: current_term: 1 leader_uuid: "c2f8d76faa67494ba8ca28c99feffb7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f8d76faa67494ba8ca28c99feffb7c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43577 } } }
I20260926 14:50:59.797786 22350 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:59.797868 22357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:59.800761 22357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:59.805680 22357 catalog_manager.cc:1347] Generated new cluster ID: 7c790c01f21049c4ab4fb6738a7912d4
I20260926 14:50:59.805747 22357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:59.831611 22357 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:59.831954 22357 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:59.849274 22357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2f8d76faa67494ba8ca28c99feffb7c: Generated new TSK 0
I20260926 14:50:59.850013 22357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:50:59.860250 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43577
--builtin_ntp_servers=127.12.68.212:35431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 14:50:59.963828 22367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:59.964057 22367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:59.964097 22367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:59.967984 22367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:59.968029 22367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:59.968093 22367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:50:59.972641 22367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:35431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:50:59.973536 22367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:59.974799 22367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:59.983502 22372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:59.983510 22373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:59.983515 22375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:59.983790 22367 server_base.cc:1034] running on GCE node
I20260926 14:50:59.984416 22367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:59.985014 22367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:59.986173 22367 hybrid_clock.cc:648] HybridClock initialized: now 1790434259986153 us; error 31 us; skew 500 ppm
I20260926 14:50:59.987927 22367 webserver.cc:460] Webserver started at http://127.12.68.193:33043/ using document root <none> and password file <none>
I20260926 14:50:59.988449 22367 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:59.988534 22367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:59.988726 22367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:59.990648 22367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "915824abc355449db8bac9c1df90e933"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "493803fde3d021ea66d8153bdb150e45"
server_key_iv: "f98a918cfa441f7e70db2549b2a81758"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.991149 22367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "915824abc355449db8bac9c1df90e933"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "493803fde3d021ea66d8153bdb150e45"
server_key_iv: "f98a918cfa441f7e70db2549b2a81758"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.994565 22367 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 14:50:59.996968 22381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:59.998112 22367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:59.998214 22367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "915824abc355449db8bac9c1df90e933"
format_stamp: "Formatted at 2026-09-26 14:50:59 on dist-test-slave-24mn"
server_key: "493803fde3d021ea66d8153bdb150e45"
server_key_iv: "f98a918cfa441f7e70db2549b2a81758"
server_key_version: "encryptionkey@0"
I20260926 14:50:59.998302 22367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:51:00.002524 22367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:51:00.003057 22367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:51:00.003201 22367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:51:00.003677 22367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:51:00.004510 22367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:51:00.004555 22367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:00.004601 22367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:51:00.004628 22367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:00.012734 22367 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:43085
I20260926 14:51:00.012764 22494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:43085 every 8 connection(s)
I20260926 14:51:00.013526 22367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:51:00.015436 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 22367
I20260926 14:51:00.015549 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:51:00.015822 12563 external_mini_cluster.cc:1428] Setting key 631229d7c9fa0bc04cf23f11f13f246f
I20260926 14:51:00.017587 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43577
--builtin_ntp_servers=127.12.68.212:35431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 14:51:00.019461 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:00.015243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:57433 (local address 127.12.68.254:43577)
0926 14:51:00.015397 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:00.015402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:00.016495 (+ 1093us) server_negotiation.cc:407] Connection header received
0926 14:51:00.017798 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:00.017803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:00.017839 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:00.017880 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:51:00.019058 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:51:00.019061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:51:00.019064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:51:00.019076 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:51:00.019095 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:51:00.019102 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:51:00.019178 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:00.019293 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 14:51:00.019335 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:51:00.020337 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:00.015634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43577 (local address 127.12.68.193:57433)
0926 14:51:00.016348 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:00.016399 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:00.017590 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:00.018171 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:00.018177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:00.018265 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:51:00.018812 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:51:00.018814 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:51:00.018852 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:51:00.018919 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:51:00.019170 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:51:00.019177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:51:00.019182 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:51:00.019331 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 14:51:00.019565 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":101,"threads_started":1}
I20260926 14:51:00.021588 22495 heartbeater.cc:347] Connected to a master server at 127.12.68.254:43577
I20260926 14:51:00.021797 22495 heartbeater.cc:464] Registering TS with master...
I20260926 14:51:00.022189 22495 heartbeater.cc:511] Master 127.12.68.254:43577 requested a full tablet report, sending...
I20260926 14:51:00.023765 22307 ts_manager.cc:194] Registered new tserver with Master: 915824abc355449db8bac9c1df90e933 (127.12.68.193:43085)
W20260926 14:51:00.121309 22498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:51:00.121536 22498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:51:00.121584 22498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:51:00.125448 22498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:51:00.125496 22498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:51:00.125568 22498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:51:00.130116 22498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:35431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:51:00.131032 22498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:51:00.132398 22498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:51:00.141012 22506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:51:00.141100 22504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:00.141194 22498 server_base.cc:1034] running on GCE node
W20260926 14:51:00.141028 22503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:00.141620 22498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:51:00.142153 22498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:51:00.143319 22498 hybrid_clock.cc:648] HybridClock initialized: now 1790434260143301 us; error 28 us; skew 500 ppm
I20260926 14:51:00.144937 22498 webserver.cc:460] Webserver started at http://127.12.68.194:42335/ using document root <none> and password file <none>
I20260926 14:51:00.145442 22498 fs_manager.cc:359] Metadata directory not provided
I20260926 14:51:00.145503 22498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:51:00.145673 22498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:51:00.147579 22498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "6c5812fd3ad340afac8a2f58de920511"
format_stamp: "Formatted at 2026-09-26 14:51:00 on dist-test-slave-24mn"
server_key: "03dbd0b28b61ef5e2e03565bcf4c668e"
server_key_iv: "84c4ee4ce844ba1766198c1c8f5290a9"
server_key_version: "encryptionkey@0"
I20260926 14:51:00.148089 22498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "6c5812fd3ad340afac8a2f58de920511"
format_stamp: "Formatted at 2026-09-26 14:51:00 on dist-test-slave-24mn"
server_key: "03dbd0b28b61ef5e2e03565bcf4c668e"
server_key_iv: "84c4ee4ce844ba1766198c1c8f5290a9"
server_key_version: "encryptionkey@0"
I20260926 14:51:00.151341 22498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:51:00.153743 22512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:00.154855 22498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 14:51:00.154963 22498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "6c5812fd3ad340afac8a2f58de920511"
format_stamp: "Formatted at 2026-09-26 14:51:00 on dist-test-slave-24mn"
server_key: "03dbd0b28b61ef5e2e03565bcf4c668e"
server_key_iv: "84c4ee4ce844ba1766198c1c8f5290a9"
server_key_version: "encryptionkey@0"
I20260926 14:51:00.155059 22498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:51:00.159287 22498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:51:00.159816 22498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:51:00.159974 22498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:51:00.160454 22498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:51:00.161269 22498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:51:00.161314 22498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:00.161360 22498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:51:00.161386 22498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:00.169394 22498 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:42049
I20260926 14:51:00.169423 22625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:42049 every 8 connection(s)
I20260926 14:51:00.170190 22498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:51:00.172765 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 22498
I20260926 14:51:00.172880 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:51:00.173188 12563 external_mini_cluster.cc:1428] Setting key 29f1fa98a14bc57404297c71e5664ca4
I20260926 14:51:00.175176 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43577
--builtin_ntp_servers=127.12.68.212:35431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 14:51:00.176085 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:00.171770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:57163 (local address 127.12.68.254:43577)
0926 14:51:00.171904 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:00.171907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:00.173024 (+ 1117us) server_negotiation.cc:407] Connection header received
0926 14:51:00.174484 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:00.174487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:00.174519 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:00.174584 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:51:00.175656 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:51:00.175659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:51:00.175662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:51:00.175674 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:51:00.175694 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:51:00.175702 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:51:00.175776 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:00.175918 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 14:51:00.175959 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:51:00.177004 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:00.172171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43577 (local address 127.12.68.194:57163)
0926 14:51:00.172867 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:00.172919 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:00.174279 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:00.174732 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:00.174738 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:00.174823 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:51:00.175419 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:51:00.175420 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:51:00.175458 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:51:00.175527 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:51:00.175775 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:51:00.175783 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:51:00.175789 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:51:00.175953 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 14:51:00.176173 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":99,"threads_started":1}
I20260926 14:51:00.178062 22626 heartbeater.cc:347] Connected to a master server at 127.12.68.254:43577
I20260926 14:51:00.178284 22626 heartbeater.cc:464] Registering TS with master...
I20260926 14:51:00.178638 22626 heartbeater.cc:511] Master 127.12.68.254:43577 requested a full tablet report, sending...
I20260926 14:51:00.179548 22307 ts_manager.cc:194] Registered new tserver with Master: 6c5812fd3ad340afac8a2f58de920511 (127.12.68.194:42049)
W20260926 14:51:00.279770 22629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:51:00.280009 22629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:51:00.280051 22629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:51:00.283975 22629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:51:00.284016 22629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:51:00.284078 22629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:51:00.288568 22629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:35431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:43577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:51:00.289467 22629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:51:00.290720 22629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:51:00.299553 22635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:51:00.299579 22634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:00.299794 22629 server_base.cc:1034] running on GCE node
W20260926 14:51:00.299552 22637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:00.300331 22629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:51:00.300925 22629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:51:00.302096 22629 hybrid_clock.cc:648] HybridClock initialized: now 1790434260302071 us; error 33 us; skew 500 ppm
I20260926 14:51:00.303630 22629 webserver.cc:460] Webserver started at http://127.12.68.195:38983/ using document root <none> and password file <none>
I20260926 14:51:00.304216 22629 fs_manager.cc:359] Metadata directory not provided
I20260926 14:51:00.304275 22629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:51:00.304451 22629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:51:00.306309 22629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "642c11eee2774b74a1e8d1953d37a7fc"
format_stamp: "Formatted at 2026-09-26 14:51:00 on dist-test-slave-24mn"
server_key: "4d3cdf19bb759deb683b3f12e4bbb26f"
server_key_iv: "aa680d53efe8384ccff413841606e6b3"
server_key_version: "encryptionkey@0"
I20260926 14:51:00.306805 22629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "642c11eee2774b74a1e8d1953d37a7fc"
format_stamp: "Formatted at 2026-09-26 14:51:00 on dist-test-slave-24mn"
server_key: "4d3cdf19bb759deb683b3f12e4bbb26f"
server_key_iv: "aa680d53efe8384ccff413841606e6b3"
server_key_version: "encryptionkey@0"
I20260926 14:51:00.310181 22629 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:51:00.312429 22643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:00.313484 22629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:51:00.313571 22629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "642c11eee2774b74a1e8d1953d37a7fc"
format_stamp: "Formatted at 2026-09-26 14:51:00 on dist-test-slave-24mn"
server_key: "4d3cdf19bb759deb683b3f12e4bbb26f"
server_key_iv: "aa680d53efe8384ccff413841606e6b3"
server_key_version: "encryptionkey@0"
I20260926 14:51:00.313666 22629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:51:00.317785 22629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:51:00.318302 22629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:51:00.318444 22629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:51:00.318903 22629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:51:00.319684 22629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:51:00.319727 22629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:00.319761 22629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:51:00.319787 22629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:00.327549 22629 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:44927
I20260926 14:51:00.327598 22756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:44927 every 8 connection(s)
I20260926 14:51:00.328342 22629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:51:00.330483 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 22629
I20260926 14:51:00.330600 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:51:00.330915 12563 external_mini_cluster.cc:1428] Setting key 6716f533915fb7c142111538ce919845
I20260926 14:51:00.334000 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:00.329958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:52085 (local address 127.12.68.254:43577)
0926 14:51:00.330086 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:00.330090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:00.331134 (+ 1044us) server_negotiation.cc:407] Connection header received
0926 14:51:00.332510 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:00.332514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:00.332554 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:00.332623 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:51:00.333601 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:51:00.333604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:51:00.333607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:51:00.333619 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:51:00.333638 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:51:00.333646 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:51:00.333715 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:00.333840 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 14:51:00.333881 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:51:00.334746 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:00.330326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43577 (local address 127.12.68.195:52085)
0926 14:51:00.331000 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:00.331044 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:00.332321 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:00.332789 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:00.332795 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:00.332878 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:51:00.333360 (+ 482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:51:00.333361 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:51:00.333398 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:51:00.333461 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:51:00.333714 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:51:00.333721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:51:00.333726 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:51:00.333868 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 14:51:00.334087 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":123,"threads_started":1}
I20260926 14:51:00.335628 22757 heartbeater.cc:347] Connected to a master server at 127.12.68.254:43577
I20260926 14:51:00.335812 22757 heartbeater.cc:464] Registering TS with master...
I20260926 14:51:00.336179 22757 heartbeater.cc:511] Master 127.12.68.254:43577 requested a full tablet report, sending...
I20260926 14:51:00.337021 22307 ts_manager.cc:194] Registered new tserver with Master: 642c11eee2774b74a1e8d1953d37a7fc (127.12.68.195:44927)
I20260926 14:51:00.344228 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 14:51:00.346630 22346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:51:00.345922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.12.68.254:43577)
0926 14:51:00.346032 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:00.346036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:00.346188 (+ 152us) server_negotiation.cc:407] Connection header received
0926 14:51:00.346273 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:00.346275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:00.346304 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:00.346358 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:51:00.346525 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60638: BlockingRecv error: recv got EOF from 127.0.0.1:60638 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:51:00.346755 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 22367
I20260926 14:51:00.351578 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 22498
I20260926 14:51:00.355952 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 22629
I20260926 14:51:00.360020 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 22276
2026-09-26T14:51:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (755 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:51:00 dist-test-slave-24mn krb5kdc[22767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:51:00 dist-test-slave-24mn krb5kdc[22767](info): set up 2 sockets
Sep 26 14:51:00 dist-test-slave-24mn krb5kdc[22767](info): commencing operation
krb5kdc: starting...
W20260926 14:51:01.308394 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:51:01 dist-test-slave-24mn krb5kdc[22767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:51:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:51:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:51:01.416723 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:33363
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:33363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:51:01.519024 22780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:51:01.519261 22780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:51:01.519297 22780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:51:01.523224 22780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:51:01.523276 22780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:51:01.523294 22780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:51:01.523310 22780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:51:01.523324 22780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:51:01.529102 22780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:33363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:51:01.530046 22780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:51:01.530974 22780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:51:01.538867 22785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:51:01.538882 22786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:01.538973 22780 server_base.cc:1034] running on GCE node
W20260926 14:51:01.538882 22788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:01.539597 22780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:51:01.540690 22780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:51:01.541848 22780 hybrid_clock.cc:648] HybridClock initialized: now 1790434261541824 us; error 35 us; skew 500 ppm
Sep 26 14:51:01 dist-test-slave-24mn krb5kdc[22767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434261, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:51:01.544384 22780 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:51:01.545154 22780 webserver.cc:460] Webserver started at http://127.12.68.254:41523/ using document root <none> and password file <none>
I20260926 14:51:01.545632 22780 fs_manager.cc:359] Metadata directory not provided
I20260926 14:51:01.545686 22780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:51:01.545866 22780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:51:01.547760 22780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "15984262a9a3460fb25f35538d495254"
format_stamp: "Formatted at 2026-09-26 14:51:01 on dist-test-slave-24mn"
server_key: "a126582cab3a8e5a87849f8291326be9"
server_key_iv: "0d5cc37e9049595b51c3a71c41eb22d9"
server_key_version: "encryptionkey@0"
I20260926 14:51:01.548305 22780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "15984262a9a3460fb25f35538d495254"
format_stamp: "Formatted at 2026-09-26 14:51:01 on dist-test-slave-24mn"
server_key: "a126582cab3a8e5a87849f8291326be9"
server_key_iv: "0d5cc37e9049595b51c3a71c41eb22d9"
server_key_version: "encryptionkey@0"
I20260926 14:51:01.551723 22780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:51:01.553917 22795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:01.555022 22780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:51:01.555123 22780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "15984262a9a3460fb25f35538d495254"
format_stamp: "Formatted at 2026-09-26 14:51:01 on dist-test-slave-24mn"
server_key: "a126582cab3a8e5a87849f8291326be9"
server_key_iv: "0d5cc37e9049595b51c3a71c41eb22d9"
server_key_version: "encryptionkey@0"
I20260926 14:51:01.555213 22780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:51:01.566259 22780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:51:01.568698 22780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:51:01.568845 22780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:51:01.574747 22780 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:33363
I20260926 14:51:01.575008 22847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:33363 every 8 connection(s)
I20260926 14:51:01.575583 22780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:51:01.578228 22848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:51:01.581501 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 22780
I20260926 14:51:01.581624 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:51:01.581907 12563 external_mini_cluster.cc:1428] Setting key 8b0c72068110a470adaeb5a8bb1841c3
I20260926 14:51:01.584167 22848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254: Bootstrap starting.
I20260926 14:51:01.586495 22848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254: Neither blocks nor log segments found. Creating new log.
I20260926 14:51:01.587328 22848 log.cc:824] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254: Log is configured to *not* fsync() on all Append() calls
I20260926 14:51:01.589365 22848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254: No bootstrap required, opened a new log
Sep 26 14:51:01 dist-test-slave-24mn krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:51:01.593155 22848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15984262a9a3460fb25f35538d495254" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33363 } }
I20260926 14:51:01.593431 22848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:51:01.593489 22848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15984262a9a3460fb25f35538d495254, State: Initialized, Role: FOLLOWER
I20260926 14:51:01.593916 22848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15984262a9a3460fb25f35538d495254" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33363 } }
I20260926 14:51:01.594044 22848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:51:01.594106 22848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:51:01.594194 22848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:51:01.595089 22848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15984262a9a3460fb25f35538d495254" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33363 } }
I20260926 14:51:01.595439 22848 leader_election.cc:302] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15984262a9a3460fb25f35538d495254; no voters:
I20260926 14:51:01.595718 22848 leader_election.cc:288] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:51:01.595814 22853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:51:01.596031 22853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [term 1 LEADER]: Becoming Leader. State: Replica: 15984262a9a3460fb25f35538d495254, State: Running, Role: LEADER
I20260926 14:51:01.596247 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:01.583340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53122 (local address 127.12.68.254:33363)
0926 14:51:01.583773 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:01.583793 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:01.583830 (+ 37us) server_negotiation.cc:407] Connection header received
0926 14:51:01.584571 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:01.584594 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:01.584938 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:01.585241 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:51:01.586081 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:01.587547 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:01.588527 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:01.589163 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:01.589317 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:01.591505 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:51:01.591511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:51:01.591526 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:51:01.591555 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:51:01.593370 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:01.593817 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:01.593823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:01.593826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:01.593941 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:01.594231 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:01.594234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:01.594235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:01.594626 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:51:01.594777 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:01.595070 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 14:51:01.595342 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":125,"threads_started":1}
I20260926 14:51:01.596333 22853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15984262a9a3460fb25f35538d495254" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33363 } }
I20260926 14:51:01.596871 22848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:51:01.597432 22853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15984262a9a3460fb25f35538d495254. Latest consensus state: current_term: 1 leader_uuid: "15984262a9a3460fb25f35538d495254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15984262a9a3460fb25f35538d495254" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33363 } } }
I20260926 14:51:01.597536 22853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [sys.catalog]: This master's current role is: LEADER
I20260926 14:51:01.597880 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15984262a9a3460fb25f35538d495254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15984262a9a3460fb25f35538d495254" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 33363 } } }
I20260926 14:51:01.597950 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254 [sys.catalog]: This master's current role is: LEADER
W20260926 14:51:01.601011 22868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:51:01.601073 22868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:51:01.601171 22860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:51:01.601804 22860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:51:01.606760 22860 catalog_manager.cc:1347] Generated new cluster ID: dc4224f09d024afa9ac3d3d62ad62e88
I20260926 14:51:01.606817 22860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:51:01.621686 22860 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:51:01.622514 22860 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:51:01.634019 22860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254: Generated new TSK 0
I20260926 14:51:01.634614 22860 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:51:01.660234 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33363
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:51:01.762593 22874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:51:01.762816 22874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:51:01.762871 22874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:51:01.766798 22874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:51:01.766849 22874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:51:01.766923 22874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:51:01.771837 22874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:51:01.772739 22874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:51:01.773645 22874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:51:01.782194 22879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:51:01.782181 22880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:01.782322 22874 server_base.cc:1034] running on GCE node
W20260926 14:51:01.782181 22882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:01.782738 22874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:51:01.783273 22874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:51:01.784406 22874 hybrid_clock.cc:648] HybridClock initialized: now 1790434261784383 us; error 35 us; skew 500 ppm
Sep 26 14:51:01 dist-test-slave-24mn krb5kdc[22767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434261, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:51:01.786792 22874 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:51:01.787547 22874 webserver.cc:460] Webserver started at http://127.12.68.193:33315/ using document root <none> and password file <none>
I20260926 14:51:01.788064 22874 fs_manager.cc:359] Metadata directory not provided
I20260926 14:51:01.788144 22874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:51:01.788329 22874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:51:01.790148 22874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "b832a6d2c4b149b2998df27262a5efb9"
format_stamp: "Formatted at 2026-09-26 14:51:01 on dist-test-slave-24mn"
server_key: "571af8864090df7ee79cffd77acae3c9"
server_key_iv: "3c1eda92bc005363367a83f4e0b6868c"
server_key_version: "encryptionkey@0"
I20260926 14:51:01.790635 22874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "b832a6d2c4b149b2998df27262a5efb9"
format_stamp: "Formatted at 2026-09-26 14:51:01 on dist-test-slave-24mn"
server_key: "571af8864090df7ee79cffd77acae3c9"
server_key_iv: "3c1eda92bc005363367a83f4e0b6868c"
server_key_version: "encryptionkey@0"
I20260926 14:51:01.793905 22874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:51:01.795984 22889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:01.797063 22874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:51:01.797174 22874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "b832a6d2c4b149b2998df27262a5efb9"
format_stamp: "Formatted at 2026-09-26 14:51:01 on dist-test-slave-24mn"
server_key: "571af8864090df7ee79cffd77acae3c9"
server_key_iv: "3c1eda92bc005363367a83f4e0b6868c"
server_key_version: "encryptionkey@0"
I20260926 14:51:01.797273 22874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:51:01.810205 22874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:51:01.812640 22874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:51:01.812794 22874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:51:01.813295 22874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:51:01.814010 22874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:51:01.814054 22874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:01.814111 22874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:51:01.814138 22874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:01.821898 22874 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:37881
I20260926 14:51:01.821916 23002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:37881 every 8 connection(s)
I20260926 14:51:01.822656 22874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:51:01.825381 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 22874
I20260926 14:51:01.825477 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:51:01.825727 12563 external_mini_cluster.cc:1428] Setting key 7d30d2ac6abaf554cdb6d5fd50e0c9e3
Sep 26 14:51:01 dist-test-slave-24mn krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434261, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
I20260926 14:51:01.835948 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:01.824232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:33521 (local address 127.12.68.254:33363)
0926 14:51:01.824357 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:01.824361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:01.825092 (+ 731us) server_negotiation.cc:407] Connection header received
0926 14:51:01.826402 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:01.826405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:01.826446 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:01.826510 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:51:01.827595 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:01.828354 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:01.829894 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:01.830227 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:01.830318 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:01.832730 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:51:01.832733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:51:01.832735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:51:01.832757 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:51:01.833973 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:01.834555 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:01.834558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:01.834559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:01.834625 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:01.834988 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:01.834990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:01.834991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:01.835165 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:51:01.835238 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:01.835673 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 14:51:01.835773 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:51:01.836625 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:01.824566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:33363 (local address 127.12.68.193:33521)
0926 14:51:01.824957 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:01.824994 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:01.826219 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:01.826668 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:01.826677 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:01.827025 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:51:01.827457 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:01.827470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:01.828486 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:01.828490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:01.829774 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:01.829785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:01.830350 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:01.830353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:01.830479 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:01.831045 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:51:01.831060 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:51:01.832568 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:51:01.834086 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:51:01.834092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:51:01.834106 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:51:01.834439 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:51:01.834700 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:51:01.834703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:51:01.834705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:51:01.834870 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:51:01.835261 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:51:01.835265 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:51:01.835466 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 14:51:01.835613 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 14:51:01.835866 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
I20260926 14:51:01.837777 23003 heartbeater.cc:347] Connected to a master server at 127.12.68.254:33363
I20260926 14:51:01.838012 23003 heartbeater.cc:464] Registering TS with master...
I20260926 14:51:01.838522 23003 heartbeater.cc:511] Master 127.12.68.254:33363 requested a full tablet report, sending...
I20260926 14:51:01.840045 22812 ts_manager.cc:194] Registered new tserver with Master: b832a6d2c4b149b2998df27262a5efb9 (127.12.68.193:37881)
I20260926 14:51:01.841096 22812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:33521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:51:01.850358 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33363
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:51:01.953230 23008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:51:01.953439 23008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:51:01.953477 23008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:51:01.957330 23008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:51:01.957378 23008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:51:01.957429 23008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:51:01.962355 23008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:51:01.963197 23008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:51:01.964069 23008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:51:01.972762 23016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:01.972810 23008 server_base.cc:1034] running on GCE node
W20260926 14:51:01.972790 23013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:51:01.972767 23014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:01.973340 23008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:51:01.973939 23008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:51:01.975085 23008 hybrid_clock.cc:648] HybridClock initialized: now 1790434261975065 us; error 36 us; skew 500 ppm
Sep 26 14:51:01 dist-test-slave-24mn krb5kdc[22767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434261, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:51:01.977603 23008 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:51:01.978369 23008 webserver.cc:460] Webserver started at http://127.12.68.194:42807/ using document root <none> and password file <none>
I20260926 14:51:01.978991 23008 fs_manager.cc:359] Metadata directory not provided
I20260926 14:51:01.979056 23008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:51:01.979218 23008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:51:01.981133 23008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "cceb107eba404e5c9dee0a8e0631a685"
format_stamp: "Formatted at 2026-09-26 14:51:01 on dist-test-slave-24mn"
server_key: "b3d28e34ac9ac18f42130540dc6a56b4"
server_key_iv: "2951823b86b00c601a94530df7f18810"
server_key_version: "encryptionkey@0"
I20260926 14:51:01.981609 23008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "cceb107eba404e5c9dee0a8e0631a685"
format_stamp: "Formatted at 2026-09-26 14:51:01 on dist-test-slave-24mn"
server_key: "b3d28e34ac9ac18f42130540dc6a56b4"
server_key_iv: "2951823b86b00c601a94530df7f18810"
server_key_version: "encryptionkey@0"
I20260926 14:51:01.984978 23008 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:51:01.987185 23023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:01.988333 23008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:51:01.988433 23008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "cceb107eba404e5c9dee0a8e0631a685"
format_stamp: "Formatted at 2026-09-26 14:51:01 on dist-test-slave-24mn"
server_key: "b3d28e34ac9ac18f42130540dc6a56b4"
server_key_iv: "2951823b86b00c601a94530df7f18810"
server_key_version: "encryptionkey@0"
I20260926 14:51:01.988533 23008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:51:02.003911 23008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:51:02.006461 23008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:51:02.006613 23008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:51:02.007162 23008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:51:02.007982 23008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:51:02.008026 23008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:02.008066 23008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:51:02.008093 23008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:02.015776 23008 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:40087
I20260926 14:51:02.015800 23136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:40087 every 8 connection(s)
I20260926 14:51:02.016578 23008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:51:02 dist-test-slave-24mn krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434261, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:51:02.025481 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 23008
I20260926 14:51:02.025580 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:51:02.025876 12563 external_mini_cluster.cc:1428] Setting key 99f8a41e86b0eba568392f6af6407c9e
I20260926 14:51:02.028882 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.018160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:46399 (local address 127.12.68.254:33363)
0926 14:51:02.018313 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:02.018318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:02.019065 (+ 747us) server_negotiation.cc:407] Connection header received
0926 14:51:02.020379 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:02.020382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:02.020429 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:02.020501 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:51:02.021565 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.022084 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.023186 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.023392 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.023457 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.025705 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:51:02.025709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:51:02.025712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:51:02.025740 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:51:02.027069 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:02.027502 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:02.027505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:02.027507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:02.027597 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:02.027910 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:02.027913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:02.027915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:02.028111 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:51:02.028201 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:02.028609 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 14:51:02.028742 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:51:02.029898 23139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.018525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:33363 (local address 127.12.68.194:46399)
0926 14:51:02.018928 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:02.018967 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:02.020183 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:02.020641 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:02.020647 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:02.020971 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:51:02.021451 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:02.021464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.022211 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.022214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:02.023083 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:02.023092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.023529 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.023532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:02.023674 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.024169 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:51:02.024184 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:51:02.025515 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:51:02.027170 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:51:02.027175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:51:02.027186 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:51:02.027412 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:51:02.027693 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:51:02.027695 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:51:02.027696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:51:02.027812 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:51:02.028194 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:51:02.028198 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:51:02.028400 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 14:51:02.028557 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 14:51:02.028868 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":116,"threads_started":1}
I20260926 14:51:02.030915 23137 heartbeater.cc:347] Connected to a master server at 127.12.68.254:33363
I20260926 14:51:02.031126 23137 heartbeater.cc:464] Registering TS with master...
I20260926 14:51:02.031548 23137 heartbeater.cc:511] Master 127.12.68.254:33363 requested a full tablet report, sending...
I20260926 14:51:02.032501 22812 ts_manager.cc:194] Registered new tserver with Master: cceb107eba404e5c9dee0a8e0631a685 (127.12.68.194:40087)
I20260926 14:51:02.033018 22812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:46399
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:51:02.050220 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33363
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:51:02.153087 23142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:51:02.153373 23142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:51:02.153432 23142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:51:02.157779 23142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:51:02.157831 23142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:51:02.157896 23142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:51:02.163012 23142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:33363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:51:02.163966 23142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:51:02.164891 23142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:51:02.173830 23147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:51:02.173832 23148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:02.173911 23142 server_base.cc:1034] running on GCE node
W20260926 14:51:02.173830 23150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:02.174571 23142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:51:02.175149 23142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:51:02.176295 23142 hybrid_clock.cc:648] HybridClock initialized: now 1790434262176279 us; error 31 us; skew 500 ppm
Sep 26 14:51:02 dist-test-slave-24mn krb5kdc[22767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434262, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:51:02.178854 23142 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:51:02.179597 23142 webserver.cc:460] Webserver started at http://127.12.68.195:44567/ using document root <none> and password file <none>
I20260926 14:51:02.180128 23142 fs_manager.cc:359] Metadata directory not provided
I20260926 14:51:02.180186 23142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:51:02.180347 23142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:51:02.182204 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "a19ed044e86b4c8d881045228cebe6a5"
format_stamp: "Formatted at 2026-09-26 14:51:02 on dist-test-slave-24mn"
server_key: "f694635cce552bdc8ba63c52e891f5c9"
server_key_iv: "767a7a872cd6fadcba61410319450f3d"
server_key_version: "encryptionkey@0"
I20260926 14:51:02.182687 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "a19ed044e86b4c8d881045228cebe6a5"
format_stamp: "Formatted at 2026-09-26 14:51:02 on dist-test-slave-24mn"
server_key: "f694635cce552bdc8ba63c52e891f5c9"
server_key_iv: "767a7a872cd6fadcba61410319450f3d"
server_key_version: "encryptionkey@0"
I20260926 14:51:02.186100 23142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:51:02.188287 23157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:02.189376 23142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:51:02.189476 23142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "a19ed044e86b4c8d881045228cebe6a5"
format_stamp: "Formatted at 2026-09-26 14:51:02 on dist-test-slave-24mn"
server_key: "f694635cce552bdc8ba63c52e891f5c9"
server_key_iv: "767a7a872cd6fadcba61410319450f3d"
server_key_version: "encryptionkey@0"
I20260926 14:51:02.189559 23142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:51:02.211978 23142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:51:02.214622 23142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:51:02.214797 23142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:51:02.215349 23142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:51:02.216136 23142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:51:02.216182 23142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:02.216241 23142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:51:02.216269 23142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:02.224159 23142 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:36013
I20260926 14:51:02.224190 23270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:36013 every 8 connection(s)
I20260926 14:51:02.225014 23142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
Sep 26 14:51:02 dist-test-slave-24mn krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434262, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:51:02.235275 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 23142
I20260926 14:51:02.235373 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:51:02.235626 12563 external_mini_cluster.cc:1428] Setting key dcbe4976e47f01f6a18c1678c2bbdfe3
I20260926 14:51:02.237778 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.226703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:45869 (local address 127.12.68.254:33363)
0926 14:51:02.226873 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:02.226877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:02.227670 (+ 793us) server_negotiation.cc:407] Connection header received
0926 14:51:02.229043 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:02.229046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:02.229091 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:02.229162 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:51:02.230260 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.230801 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.231896 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.232145 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.232233 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.234571 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:51:02.234574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:51:02.234576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:51:02.234606 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:51:02.235932 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:02.236398 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:02.236400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:02.236401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:02.236470 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:02.236792 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:02.236795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:02.236796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:02.236966 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:51:02.237044 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:02.237564 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 14:51:02.237665 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 14:51:02.238515 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.227075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:33363 (local address 127.12.68.195:45869)
0926 14:51:02.227521 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:02.227561 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:02.228846 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:02.229307 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:02.229313 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:02.229647 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:51:02.230127 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:02.230139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.230891 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.230894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:02.231747 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:02.231755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.232295 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.232298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:02.232430 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.232905 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:51:02.232921 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:51:02.234403 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:51:02.236042 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:51:02.236048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:51:02.236070 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:51:02.236311 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:51:02.236553 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:51:02.236556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:51:02.236558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:51:02.236697 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:51:02.237063 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:51:02.237069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:51:02.237347 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 14:51:02.237501 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 14:51:02.237768 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":120,"threads_started":1}
I20260926 14:51:02.239493 23271 heartbeater.cc:347] Connected to a master server at 127.12.68.254:33363
I20260926 14:51:02.239681 23271 heartbeater.cc:464] Registering TS with master...
I20260926 14:51:02.240124 23271 heartbeater.cc:511] Master 127.12.68.254:33363 requested a full tablet report, sending...
I20260926 14:51:02.241004 22812 ts_manager.cc:194] Registered new tserver with Master: a19ed044e86b4c8d881045228cebe6a5 (127.12.68.195:36013)
I20260926 14:51:02.241508 22812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:45869
I20260926 14:51:02.249020 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:51:02.257220 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.250733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.12.68.254:33363)
0926 14:51:02.250842 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:02.250847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:02.251014 (+ 167us) server_negotiation.cc:407] Connection header received
0926 14:51:02.251120 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:02.251123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:02.251162 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:02.251233 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:51:02.251859 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.252384 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.253185 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.253398 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.253478 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.254653 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:51:02.254656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:51:02.254659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:51:02.254683 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:51:02.255723 (+ 1040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:02.256141 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:02.256145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:02.256146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:02.256223 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:02.256450 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:02.256453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:02.256455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:02.256650 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:51:02.256764 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:02.256972 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 14:51:02.257069 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:51:02.259600 22812 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:51:02.262470 22812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:51:02.275825 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.270754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:40087 (local address 127.0.0.1:34554)
0926 14:51:02.271325 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:02.271337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:02.271415 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:02.272223 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:02.272226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:02.272242 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:51:02.272329 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:02.272335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.273330 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.273333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:02.274480 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:02.274488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.275332 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.275336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:02.275431 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.275463 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 14:51:02.275565 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:51:02.275670 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":45,"thread_start_us":95,"threads_started":1}
I20260926 14:51:02.275837 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.270913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.12.68.194:40087)
0926 14:51:02.271859 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:02.271870 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:02.271906 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:51:02.271975 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:02.271979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:02.272106 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:02.272235 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:51:02.272444 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.273206 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.274581 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.275217 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.275317 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.275449 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:02.275599 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 14:51:02.275692 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":141,"threads_started":1}
I20260926 14:51:02.275858 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.270081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:36013 (local address 127.0.0.1:37854)
0926 14:51:02.270770 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:02.270818 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:02.270940 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:02.271664 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:02.271669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:02.271700 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:51:02.271819 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:02.271825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.273195 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.273198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:02.274493 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:02.274504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.275554 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.275558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:02.275633 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.275651 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:51:02.275708 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:51:02.275765 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":62,"thread_start_us":60,"threads_started":1}
I20260926 14:51:02.276192 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.270238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37854 (local address 127.12.68.195:36013)
0926 14:51:02.270544 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:02.270552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:02.271060 (+ 508us) server_negotiation.cc:407] Connection header received
0926 14:51:02.271139 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:02.271144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:02.271352 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:02.271497 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:51:02.271952 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.273056 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.274620 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.275275 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.275368 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.275497 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:02.275958 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 14:51:02.276049 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":134,"threads_started":1}
I20260926 14:51:02.276398 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.271061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:37881 (local address 127.0.0.1:43344)
0926 14:51:02.271563 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:02.271574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:02.271650 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:02.272397 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:02.272400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:02.272415 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:51:02.272480 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:02.272486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.273723 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.273727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:02.274901 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:02.274910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.276073 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.276077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:02.276142 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.276158 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 14:51:02.276217 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 14:51:02.276279 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":92,"thread_start_us":77,"threads_started":1}
I20260926 14:51:02.276780 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.271219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.12.68.193:37881)
0926 14:51:02.271483 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:02.271494 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:02.271953 (+ 459us) server_negotiation.cc:407] Connection header received
0926 14:51:02.272031 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:02.272036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:02.272240 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:02.272376 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:51:02.272570 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.273589 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.275017 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.275771 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.275866 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.276015 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:02.276546 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 14:51:02.276634 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":80,"threads_started":1}
I20260926 14:51:02.278929 22937 tablet_service.cc:1467] Processing CreateTablet for tablet d6018d916b2b431e8cf43b3155e51732 (DEFAULT_TABLE table=test-table [id=7e961a370aeb4bccaae6d62979fc2c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:51:02.278976 23205 tablet_service.cc:1467] Processing CreateTablet for tablet d6018d916b2b431e8cf43b3155e51732 (DEFAULT_TABLE table=test-table [id=7e961a370aeb4bccaae6d62979fc2c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:51:02.278976 23071 tablet_service.cc:1467] Processing CreateTablet for tablet d6018d916b2b431e8cf43b3155e51732 (DEFAULT_TABLE table=test-table [id=7e961a370aeb4bccaae6d62979fc2c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:51:02.279846 22937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6018d916b2b431e8cf43b3155e51732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:51:02.279963 23205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6018d916b2b431e8cf43b3155e51732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:51:02.280064 23071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6018d916b2b431e8cf43b3155e51732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:51:02.285522 23288 tablet_bootstrap.cc:492] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9: Bootstrap starting.
I20260926 14:51:02.286749 23289 tablet_bootstrap.cc:492] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5: Bootstrap starting.
I20260926 14:51:02.288190 23290 tablet_bootstrap.cc:492] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685: Bootstrap starting.
I20260926 14:51:02.288733 23289 tablet_bootstrap.cc:654] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5: Neither blocks nor log segments found. Creating new log.
I20260926 14:51:02.288504 23288 tablet_bootstrap.cc:654] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9: Neither blocks nor log segments found. Creating new log.
I20260926 14:51:02.289707 23288 log.cc:824] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:51:02.289839 23289 log.cc:824] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5: Log is configured to *not* fsync() on all Append() calls
I20260926 14:51:02.290202 23290 tablet_bootstrap.cc:654] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685: Neither blocks nor log segments found. Creating new log.
I20260926 14:51:02.290858 23290 log.cc:824] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685: Log is configured to *not* fsync() on all Append() calls
I20260926 14:51:02.291647 23289 tablet_bootstrap.cc:492] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5: No bootstrap required, opened a new log
I20260926 14:51:02.291646 23288 tablet_bootstrap.cc:492] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9: No bootstrap required, opened a new log
I20260926 14:51:02.291824 23289 ts_tablet_manager.cc:1397] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:51:02.291824 23288 ts_tablet_manager.cc:1397] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260926 14:51:02.292420 23290 tablet_bootstrap.cc:492] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685: No bootstrap required, opened a new log
I20260926 14:51:02.292577 23290 ts_tablet_manager.cc:1397] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260926 14:51:02.295428 23290 raft_consensus.cc:348] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19ed044e86b4c8d881045228cebe6a5" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36013 } } peers { permanent_uuid: "cceb107eba404e5c9dee0a8e0631a685" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 40087 } } peers { permanent_uuid: "b832a6d2c4b149b2998df27262a5efb9" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37881 } }
I20260926 14:51:02.295454 23289 raft_consensus.cc:348] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19ed044e86b4c8d881045228cebe6a5" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36013 } } peers { permanent_uuid: "cceb107eba404e5c9dee0a8e0631a685" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 40087 } } peers { permanent_uuid: "b832a6d2c4b149b2998df27262a5efb9" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37881 } }
I20260926 14:51:02.295660 23290 raft_consensus.cc:374] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:51:02.295679 23289 raft_consensus.cc:374] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:51:02.295712 23290 raft_consensus.cc:729] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cceb107eba404e5c9dee0a8e0631a685, State: Initialized, Role: FOLLOWER
I20260926 14:51:02.295727 23289 raft_consensus.cc:729] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a19ed044e86b4c8d881045228cebe6a5, State: Initialized, Role: FOLLOWER
I20260926 14:51:02.295827 23288 raft_consensus.cc:348] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19ed044e86b4c8d881045228cebe6a5" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36013 } } peers { permanent_uuid: "cceb107eba404e5c9dee0a8e0631a685" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 40087 } } peers { permanent_uuid: "b832a6d2c4b149b2998df27262a5efb9" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37881 } }
I20260926 14:51:02.296054 23288 raft_consensus.cc:374] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:51:02.296106 23288 raft_consensus.cc:729] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b832a6d2c4b149b2998df27262a5efb9, State: Initialized, Role: FOLLOWER
I20260926 14:51:02.296125 23290 consensus_queue.cc:260] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19ed044e86b4c8d881045228cebe6a5" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36013 } } peers { permanent_uuid: "cceb107eba404e5c9dee0a8e0631a685" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 40087 } } peers { permanent_uuid: "b832a6d2c4b149b2998df27262a5efb9" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37881 } }
I20260926 14:51:02.296149 23289 consensus_queue.cc:260] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19ed044e86b4c8d881045228cebe6a5" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36013 } } peers { permanent_uuid: "cceb107eba404e5c9dee0a8e0631a685" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 40087 } } peers { permanent_uuid: "b832a6d2c4b149b2998df27262a5efb9" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37881 } }
I20260926 14:51:02.296521 23288 consensus_queue.cc:260] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19ed044e86b4c8d881045228cebe6a5" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36013 } } peers { permanent_uuid: "cceb107eba404e5c9dee0a8e0631a685" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 40087 } } peers { permanent_uuid: "b832a6d2c4b149b2998df27262a5efb9" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37881 } }
I20260926 14:51:02.296700 23137 heartbeater.cc:503] Master 127.12.68.254:33363 was elected leader, sending a full tablet report...
I20260926 14:51:02.297066 23289 ts_tablet_manager.cc:1428] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260926 14:51:02.297279 23288 ts_tablet_manager.cc:1428] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:51:02.297453 23271 heartbeater.cc:503] Master 127.12.68.254:33363 was elected leader, sending a full tablet report...
I20260926 14:51:02.297580 23003 heartbeater.cc:503] Master 127.12.68.254:33363 was elected leader, sending a full tablet report...
I20260926 14:51:02.298249 23290 ts_tablet_manager.cc:1428] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
W20260926 14:51:02.323614 23004 tablet.cc:2369] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:51:02.377369 23294 raft_consensus.cc:482] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:51:02.377566 23294 raft_consensus.cc:504] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19ed044e86b4c8d881045228cebe6a5" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36013 } } peers { permanent_uuid: "cceb107eba404e5c9dee0a8e0631a685" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 40087 } } peers { permanent_uuid: "b832a6d2c4b149b2998df27262a5efb9" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37881 } }
I20260926 14:51:02.378659 23294 leader_election.cc:288] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a19ed044e86b4c8d881045228cebe6a5 (127.12.68.195:36013), b832a6d2c4b149b2998df27262a5efb9 (127.12.68.193:37881)
I20260926 14:51:02.382140 23139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.378893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:36013 (local address 127.12.68.194:43907)
0926 14:51:02.379015 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:02.379030 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:02.379132 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:02.379461 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:02.379464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:02.379482 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:51:02.379576 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:02.379581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.380350 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.380354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:02.381146 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:02.381154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.381750 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.381753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:02.381853 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.381874 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:51:02.381939 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:51:02.382007 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
I20260926 14:51:02.382139 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.378958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:43907 (local address 127.12.68.195:36013)
0926 14:51:02.379087 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:02.379091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:02.379108 (+ 17us) server_negotiation.cc:407] Connection header received
0926 14:51:02.379256 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:02.379259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:02.379309 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:02.379375 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:51:02.379681 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.380253 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.381263 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.381651 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.381750 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.381800 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:02.381964 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 14:51:02.382025 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:51:02.382339 23297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.379143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:37881 (local address 127.12.68.194:35911)
0926 14:51:02.379473 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:02.379484 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:02.379577 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:02.379799 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:02.379802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:02.379816 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:51:02.379950 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:02.379957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.380607 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.380611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:02.381356 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:02.381364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.381947 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.381950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:02.382054 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.382074 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:51:02.382135 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:51:02.382201 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":108,"threads_started":1}
I20260926 14:51:02.382328 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.379276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:35911 (local address 127.12.68.193:37881)
0926 14:51:02.379431 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:02.379435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:02.379549 (+ 114us) server_negotiation.cc:407] Connection header received
0926 14:51:02.379675 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:02.379679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:02.379727 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:02.379782 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:51:02.380047 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.380512 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.381471 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.381845 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.381945 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.381993 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:02.382157 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 14:51:02.382217 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 14:51:02.382761 23225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6018d916b2b431e8cf43b3155e51732" candidate_uuid: "cceb107eba404e5c9dee0a8e0631a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a19ed044e86b4c8d881045228cebe6a5" is_pre_election: true
I20260926 14:51:02.382925 22957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6018d916b2b431e8cf43b3155e51732" candidate_uuid: "cceb107eba404e5c9dee0a8e0631a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b832a6d2c4b149b2998df27262a5efb9" is_pre_election: true
I20260926 14:51:02.383033 23225 raft_consensus.cc:2454] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cceb107eba404e5c9dee0a8e0631a685 in term 0.
I20260926 14:51:02.383184 22957 raft_consensus.cc:2454] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cceb107eba404e5c9dee0a8e0631a685 in term 0.
I20260926 14:51:02.383446 23025 leader_election.cc:302] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a19ed044e86b4c8d881045228cebe6a5, cceb107eba404e5c9dee0a8e0631a685; no voters:
I20260926 14:51:02.383658 23294 raft_consensus.cc:2781] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:51:02.383731 23294 raft_consensus.cc:482] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:51:02.383760 23294 raft_consensus.cc:3037] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:51:02.384722 23294 raft_consensus.cc:504] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19ed044e86b4c8d881045228cebe6a5" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36013 } } peers { permanent_uuid: "cceb107eba404e5c9dee0a8e0631a685" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 40087 } } peers { permanent_uuid: "b832a6d2c4b149b2998df27262a5efb9" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37881 } }
I20260926 14:51:02.385013 23294 leader_election.cc:288] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [CANDIDATE]: Term 1 election: Requested vote from peers a19ed044e86b4c8d881045228cebe6a5 (127.12.68.195:36013), b832a6d2c4b149b2998df27262a5efb9 (127.12.68.193:37881)
I20260926 14:51:02.385375 22957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6018d916b2b431e8cf43b3155e51732" candidate_uuid: "cceb107eba404e5c9dee0a8e0631a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b832a6d2c4b149b2998df27262a5efb9"
I20260926 14:51:02.385397 23225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6018d916b2b431e8cf43b3155e51732" candidate_uuid: "cceb107eba404e5c9dee0a8e0631a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a19ed044e86b4c8d881045228cebe6a5"
I20260926 14:51:02.385485 23225 raft_consensus.cc:3037] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:51:02.385486 22957 raft_consensus.cc:3037] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:51:02.386479 23225 raft_consensus.cc:2454] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cceb107eba404e5c9dee0a8e0631a685 in term 1.
I20260926 14:51:02.386487 22957 raft_consensus.cc:2454] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cceb107eba404e5c9dee0a8e0631a685 in term 1.
I20260926 14:51:02.386731 23025 leader_election.cc:302] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b832a6d2c4b149b2998df27262a5efb9, cceb107eba404e5c9dee0a8e0631a685; no voters:
I20260926 14:51:02.386873 23294 raft_consensus.cc:2781] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:51:02.387050 23294 raft_consensus.cc:686] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [term 1 LEADER]: Becoming Leader. State: Replica: cceb107eba404e5c9dee0a8e0631a685, State: Running, Role: LEADER
I20260926 14:51:02.387367 23294 consensus_queue.cc:237] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19ed044e86b4c8d881045228cebe6a5" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36013 } } peers { permanent_uuid: "cceb107eba404e5c9dee0a8e0631a685" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 40087 } } peers { permanent_uuid: "b832a6d2c4b149b2998df27262a5efb9" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37881 } }
I20260926 14:51:02.390249 22811 catalog_manager.cc:5495] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to cceb107eba404e5c9dee0a8e0631a685 (127.12.68.194). New cstate: current_term: 1 leader_uuid: "cceb107eba404e5c9dee0a8e0631a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19ed044e86b4c8d881045228cebe6a5" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cceb107eba404e5c9dee0a8e0631a685" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 40087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b832a6d2c4b149b2998df27262a5efb9" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37881 } health_report { overall_health: UNKNOWN } } }
I20260926 14:51:02.426882 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.423547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34566 (local address 127.12.68.194:40087)
0926 14:51:02.423668 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:02.423672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:02.423686 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:51:02.423815 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:02.423819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:02.423873 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:02.423975 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:51:02.424200 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.424682 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.425517 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:02.425738 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:02.425822 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:02.426098 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:51:02.426490 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:51:02.426592 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:02.426672 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 14:51:02.426740 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:51:02.432200 22957 raft_consensus.cc:1261] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9 [term 1 FOLLOWER]: Refusing update from remote peer cceb107eba404e5c9dee0a8e0631a685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:51:02.432209 23225 raft_consensus.cc:1261] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5 [term 1 FOLLOWER]: Refusing update from remote peer cceb107eba404e5c9dee0a8e0631a685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:51:02.432834 23298 consensus_queue.cc:1035] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a19ed044e86b4c8d881045228cebe6a5" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:51:02.433027 23294 consensus_queue.cc:1035] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b832a6d2c4b149b2998df27262a5efb9" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:51:02.440017 23302 mvcc.cc:205] Tried to move back new op lower bound from 7333618738915676160 to 7333618738740920320. Current Snapshot: MvccSnapshot[applied={T|T < 7333618738915676160}]
I20260926 14:51:02.448410 22811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:51:02.448555 22811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:51:02.450665 22811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15984262a9a3460fb25f35538d495254: Sending DeleteTablet for 3 replicas of tablet d6018d916b2b431e8cf43b3155e51732
I20260926 14:51:02.451432 22937 tablet_service.cc:1514] Processing DeleteTablet for tablet d6018d916b2b431e8cf43b3155e51732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:51:02 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:43344
I20260926 14:51:02.451586 23205 tablet_service.cc:1514] Processing DeleteTablet for tablet d6018d916b2b431e8cf43b3155e51732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:51:02 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:37854
I20260926 14:51:02.451639 23071 tablet_service.cc:1514] Processing DeleteTablet for tablet d6018d916b2b431e8cf43b3155e51732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:51:02 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:34554
I20260926 14:51:02.452016 23313 tablet_replica.cc:330] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685: stopping tablet replica
I20260926 14:51:02.452044 23311 tablet_replica.cc:330] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9: stopping tablet replica
I20260926 14:51:02.452277 23313 raft_consensus.cc:2229] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:51:02.452302 23311 raft_consensus.cc:2229] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:51:02.452530 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 22874
I20260926 14:51:02.452561 23311 raft_consensus.cc:2258] T d6018d916b2b431e8cf43b3155e51732 P b832a6d2c4b149b2998df27262a5efb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:51:02.452600 23313 raft_consensus.cc:2258] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:51:02.453032 23312 tablet_replica.cc:330] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5: stopping tablet replica
I20260926 14:51:02.453265 23312 raft_consensus.cc:2229] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:51:02.453500 23312 raft_consensus.cc:2258] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:51:02.454675 23312 ts_tablet_manager.cc:1905] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:51:02.454737 23313 ts_tablet_manager.cc:1905] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:51:02.457068 23312 ts_tablet_manager.cc:1918] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:51:02.457146 23312 log.cc:1196] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/d6018d916b2b431e8cf43b3155e51732
I20260926 14:51:02.457407 23312 ts_tablet_manager.cc:1939] T d6018d916b2b431e8cf43b3155e51732 P a19ed044e86b4c8d881045228cebe6a5: Deleting consensus metadata
I20260926 14:51:02.458325 22797 catalog_manager.cc:4841] TS a19ed044e86b4c8d881045228cebe6a5 (127.12.68.195:36013): tablet d6018d916b2b431e8cf43b3155e51732 (table test-table [id=7e961a370aeb4bccaae6d62979fc2c8c]) successfully deleted
I20260926 14:51:02.459820 23313 ts_tablet_manager.cc:1918] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:51:02.459929 23313 log.cc:1196] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/d6018d916b2b431e8cf43b3155e51732
I20260926 14:51:02.460222 23313 ts_tablet_manager.cc:1939] T d6018d916b2b431e8cf43b3155e51732 P cceb107eba404e5c9dee0a8e0631a685: Deleting consensus metadata
I20260926 14:51:02.461162 22797 catalog_manager.cc:4841] TS cceb107eba404e5c9dee0a8e0631a685 (127.12.68.194:40087): tablet d6018d916b2b431e8cf43b3155e51732 (table test-table [id=7e961a370aeb4bccaae6d62979fc2c8c]) successfully deleted
W20260926 14:51:02.461712 22797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:37881 (error 108)
I20260926 14:51:02.462255 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 23008
I20260926 14:51:02.462899 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:02.462352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:37881 (local address 127.0.0.1:43352)
0926 14:51:02.462778 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:02.462835 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354}
W20260926 14:51:02.463104 22797 catalog_manager.cc:4584] TS b832a6d2c4b149b2998df27262a5efb9 (127.12.68.193:37881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6018d916b2b431e8cf43b3155e51732: Network error: Client connection negotiation failed: client connection to 127.12.68.193:37881: connect: Connection refused (error 111)
I20260926 14:51:02.466845 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 23142
I20260926 14:51:02.471067 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 22780
2026-09-26T14:51:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2112 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:51:02 dist-test-slave-24mn krb5kdc[23317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:51:02 dist-test-slave-24mn krb5kdc[23317](info): set up 2 sockets
Sep 26 14:51:02 dist-test-slave-24mn krb5kdc[23317](info): commencing operation
krb5kdc: starting...
W20260926 14:51:03.420482 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:51:03 dist-test-slave-24mn krb5kdc[23317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:51:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/rangeradmin_127.12.68.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/rangeradmin_127.12.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/rangerlookup_127.12.68.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/rangerlookup_127.12.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/HTTP_127.12.68.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/HTTP_127.12.68.212@KRBTEST.COM.keytab.
I20260926 14:51:03.566334 12563 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwwFvmP/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/postgres -l logfile start
2026-09-26 14:51:05.805 UTC [23347] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 14:51:05.805 UTC [23347] LOG: listening on IPv4 address "127.12.68.212", port 59293
2026-09-26 14:51:05.809 UTC [23347] LOG: listening on Unix socket "/tmp/.s.PGSQL.59293"
2026-09-26 14:51:05.836 UTC [23350] LOG: database system was shut down at 2026-09-26 14:51:04 UTC
2026-09-26 14:51:05.839 UTC [23347] LOG: database system is ready to accept connections
I20260926 14:51:06.704448 12563 mini_postgres.cc:96] Postgres bound to 59293
2026-09-26 14:51:06.710 UTC [23358] FATAL: database "slave" does not exist
127.12.68.212:59293 - accepting connections
I20260926 14:51:06.710927 12563 mini_ranger.cc:161] Starting Ranger...
I20260926 14:51:06.724468 12563 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 14:51:07.153847 12563 mini_ranger.cc:87] Created ranger Postgres database
I20260926 14:51:07.153923 12563 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-admin
2026-09-26 14:51:07,413 [I] DB FLAVOR :POSTGRES
2026-09-26 14:51:07,414 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:51:07,414 [I] Checking connection..
2026-09-26 14:51:07,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 14:51:07,636 [I] Checking connection passed.
2026-09-26 14:51:07,636 [I] --------- Verifying version history table ---------
2026-09-26 14:51:07,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:51:07,885 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:51:07,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:51:08,115 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:51:08,115 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 14:51:08,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 14:51:08.335 UTC [23422] WARNING: there is no transaction in progress
2026-09-26 14:51:08,340 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 14:51:08,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:51:08,582 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 14:51:08,583 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:51:08,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 14:51:08,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 14:51:09,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-24mn.c.gcp-upstream.internal','N') ;"
2026-09-26 14:51:09,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:51:09,508 [I] Table x_portal_user does not exist in database ranger
2026-09-26 14:51:09,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:51:09,738 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 14:51:09,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:51:09,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:51:10,211 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 14:51:10,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 14:51:11.095 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.110 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.125 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.140 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.155 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.170 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.219 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.225 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.237 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.247 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.256 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.267 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.275 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.281 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.287 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11.292 UTC [23530] WARNING: there is no transaction in progress
2026-09-26 14:51:11,872 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 14:51:11,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 14:51:12,089 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 14:51:12,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:51:12,325 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 14:51:12,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:51:12,550 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 14:51:12,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:51:12,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:51:13,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-24mn.c.gcp-upstream.internal';"
2026-09-26 14:51:13,242 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 14:51:13,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 14:51:13,488 [I] DB_PATCHES have already been applied
I20260926 14:51:13.493566 12563 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwwFvmP/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 14:51:13.493626 12563 mini_ranger.cc:193] Using host: 127.12.68.212
I20260926 14:51:13.496803 12563 mini_ranger.cc:227] Ranger admin URL: http://127.12.68.212:35723
Sep 26, 2026 2:51:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:51:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:51:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:51:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:51:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:51:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:51:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 14:51:14.498335 12563 mini_ranger.cc:160] Time spent starting Ranger: real 7.787s user 0.001s sys 0.008s
Sep 26, 2026 2:51:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:51:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 2:51:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:51:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:51:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:51:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:51:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:51:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:51:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:51:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:51:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:51:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:51:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:51:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:51:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:51:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:51:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:51:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 14:51:34.212108 12563 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/rangerkms_127.12.68.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/rangerkms_127.12.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.68.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.68.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.68.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/HTTP_127.12.68.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.68.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/HTTP_127.12.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 14:51:34 dist-test-slave-24mn krb5kdc[23317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434294, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 14:51:34.291555 12563 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 14:51:34.299945 12563 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 14:51:35.033609 12563 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 14:51:35.033689 12563 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-kms
I20260926 14:51:35.033777 12563 mini_ranger_kms.cc:227] Using postgres at 127.12.68.212:59293
2026-09-26 14:51:35,203 [I] DB FLAVOR :POSTGRES
2026-09-26 14:51:35,204 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:51:35,204 [I] Checking connection
2026-09-26 14:51:35,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 14:51:35,425 [I] connection success
2026-09-26 14:51:35,425 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 14:51:35,425 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 14:51:35,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 14:51:35,662 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 14:51:35,662 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:51:35,662 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 14:51:35,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:59293/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 14:51:35,912 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 14:51:36.336433 12563 mini_ranger_kms.cc:310] Created kms service
I20260926 14:51:36.507059 12563 mini_ranger_kms.cc:326] Created kudu user
I20260926 14:51:36.543190 12563 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 14:51:36 dist-test-slave-24mn krb5kdc[23317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434294, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:51:36.900753 12563 mini_ranger_kms.cc:384] Added ranger policy
I20260926 14:51:36.900902 12563 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 14:51:36.900929 12563 mini_ranger_kms.cc:240] Using host: 127.12.68.212
I20260926 14:51:36.904145 12563 mini_ranger_kms.cc:276] Ranger KMS PID: 23710
I20260926 14:51:36.904273 12563 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.68.212:56511
Sep 26, 2026 2:51:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:51:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:51:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:51:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:51:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:51:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 14:51:38.923305 12563 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.632s user 0.011s sys 0.143s
I20260926 14:51:38.923415 12563 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 14:51:38.923527 12563 mini_ranger_kms.cc:401] 127.12.68.212:56511/kms/v1/keys
Sep 26 14:51:39 dist-test-slave-24mn krb5kdc[23317](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434299, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.68.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 14:51:39 dist-test-slave-24mn krb5kdc[23317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434299, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.68.212@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
Sep 26 14:51:39 dist-test-slave-24mn krb5kdc[23317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434299, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.68.212@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
Sep 26 14:51:42 dist-test-slave-24mn krb5kdc[23317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:51:42.263610 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:44977
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:38611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:44977
--ranger_config_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.68.212:56511/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:51:42.367384 23761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:51:42.367640 23761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:51:42.367681 23761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:51:42.371527 23761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:51:42.371575 23761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:51:42.371593 23761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:51:42.371608 23761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:51:42.371620 23761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:51:42.376785 23761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:38611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.68.212:56511/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:44977
--ranger_config_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:44977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:51:42.377667 23761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:51:42.378769 23761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:51:42.386441 23767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:51:42.386642 23769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:42.386754 23761 server_base.cc:1034] running on GCE node
W20260926 14:51:42.386430 23766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:42.387419 23761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:51:42.388700 23761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:51:42.389878 23761 hybrid_clock.cc:648] HybridClock initialized: now 1790434302389850 us; error 33 us; skew 500 ppm
Sep 26 14:51:42 dist-test-slave-24mn krb5kdc[23317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434302, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:51:42.397017 23761 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:51:42.398029 23761 webserver.cc:460] Webserver started at http://127.12.68.254:43759/ using document root <none> and password file <none>
I20260926 14:51:42.398627 23761 fs_manager.cc:359] Metadata directory not provided
I20260926 14:51:42.398696 23761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:51:42.398922 23761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:51:42 dist-test-slave-24mn krb5kdc[23317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434302, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:51:42.442801 23761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "9e9f54a6c7cd49efbfc17dd77c989bca"
format_stamp: "Formatted at 2026-09-26 14:51:42 on dist-test-slave-24mn"
server_key: "98db25154cae30d2d1cf8edb491eb860"
server_key_iv: "7d1082073bdd1b15f2241074f9c72ce9"
server_key_version: "kuduclusterkey@0"
I20260926 14:51:42.443451 23761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "9e9f54a6c7cd49efbfc17dd77c989bca"
format_stamp: "Formatted at 2026-09-26 14:51:42 on dist-test-slave-24mn"
server_key: "98db25154cae30d2d1cf8edb491eb860"
server_key_iv: "7d1082073bdd1b15f2241074f9c72ce9"
server_key_version: "kuduclusterkey@0"
I20260926 14:51:42.448146 23761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 14:51:42.463910 23776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:42.465438 23761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:51:42.465574 23761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "9e9f54a6c7cd49efbfc17dd77c989bca"
format_stamp: "Formatted at 2026-09-26 14:51:42 on dist-test-slave-24mn"
server_key: "98db25154cae30d2d1cf8edb491eb860"
server_key_iv: "7d1082073bdd1b15f2241074f9c72ce9"
server_key_version: "kuduclusterkey@0"
I20260926 14:51:42.465694 23761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:51:42.480942 23761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:51:42.483769 23761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:51:42.483984 23761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:51:42.491737 23761 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:44977
I20260926 14:51:42.491792 23828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:44977 every 8 connection(s)
I20260926 14:51:42.492615 23761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:51:42.495301 23829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:51:42.499353 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 23761
I20260926 14:51:42.499473 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:51:42.499747 12563 external_mini_cluster.cc:1428] Setting key b2f10f3f66841af8fbe5a4f16334924a
I20260926 14:51:42.500941 23829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca: Bootstrap starting.
I20260926 14:51:42.502996 23829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca: Neither blocks nor log segments found. Creating new log.
I20260926 14:51:42.503628 23829 log.cc:824] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca: Log is configured to *not* fsync() on all Append() calls
I20260926 14:51:42.505273 23829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca: No bootstrap required, opened a new log
Sep 26 14:51:42 dist-test-slave-24mn krb5kdc[23317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:51:42.508909 23829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9f54a6c7cd49efbfc17dd77c989bca" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 44977 } }
I20260926 14:51:42.509171 23829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:51:42.509255 23829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9f54a6c7cd49efbfc17dd77c989bca, State: Initialized, Role: FOLLOWER
I20260926 14:51:42.509630 23829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9f54a6c7cd49efbfc17dd77c989bca" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 44977 } }
I20260926 14:51:42.509761 23829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:51:42.509820 23829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:51:42.509912 23829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:51:42.510596 23829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9f54a6c7cd49efbfc17dd77c989bca" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 44977 } }
I20260926 14:51:42.510892 23829 leader_election.cc:302] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e9f54a6c7cd49efbfc17dd77c989bca; no voters:
I20260926 14:51:42.511138 23829 leader_election.cc:288] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:51:42.511210 23834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:51:42.511389 23834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [term 1 LEADER]: Becoming Leader. State: Replica: 9e9f54a6c7cd49efbfc17dd77c989bca, State: Running, Role: LEADER
I20260926 14:51:42.511667 23834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9f54a6c7cd49efbfc17dd77c989bca" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 44977 } }
I20260926 14:51:42.512326 23829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:51:42.512612 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:42.501065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51272 (local address 127.12.68.254:44977)
0926 14:51:42.501452 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:42.501465 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:42.501498 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:51:42.502080 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:42.502095 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:42.502401 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:42.502684 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:51:42.503432 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:42.504544 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:42.505362 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:42.505797 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:42.505928 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:42.508340 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:51:42.508347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:51:42.508362 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:51:42.508392 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:51:42.509993 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:42.510362 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:42.510368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:42.510373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:42.510473 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:42.510704 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:42.510707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:42.510709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:42.511066 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:51:42.511230 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:42.511517 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 14:51:42.511771 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260926 14:51:42.512806 23834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e9f54a6c7cd49efbfc17dd77c989bca. Latest consensus state: current_term: 1 leader_uuid: "9e9f54a6c7cd49efbfc17dd77c989bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9f54a6c7cd49efbfc17dd77c989bca" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 44977 } } }
I20260926 14:51:42.513002 23834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [sys.catalog]: This master's current role is: LEADER
I20260926 14:51:42.513311 23835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e9f54a6c7cd49efbfc17dd77c989bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9f54a6c7cd49efbfc17dd77c989bca" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 44977 } } }
I20260926 14:51:42.513396 23842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:51:42.513468 23835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca [sys.catalog]: This master's current role is: LEADER
I20260926 14:51:42.516546 23842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:51:42.516896 23829 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 14:51:42.524726 23842 catalog_manager.cc:1347] Generated new cluster ID: 6ad3a9b42abe41328de00c93bcd13e61
I20260926 14:51:42.524793 23842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:51:42.535094 23842 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:51:42.535979 23842 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:51:42.543511 23842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca: Generated new TSK 0
I20260926 14:51:42.544116 23842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:51:43.226516 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.68.212:56511/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:44977
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:38611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:51:43.359144 23874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:51:43.359391 23874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:51:43.359449 23874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:51:43.363382 23874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:51:43.363431 23874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:51:43.363504 23874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:51:43.368702 23874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:38611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.68.212:56511/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:44977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:51:43.369964 23874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:51:43.371174 23874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:51:43.380007 23881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:51:43.380007 23884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:51:43.380422 23882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:43.381021 23874 server_base.cc:1034] running on GCE node
I20260926 14:51:43.381486 23874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:51:43.382058 23874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:51:43.385303 23874 hybrid_clock.cc:648] HybridClock initialized: now 1790434303385268 us; error 33 us; skew 500 ppm
Sep 26 14:51:43 dist-test-slave-24mn krb5kdc[23317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:51:43.388022 23874 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:51:43.388813 23874 webserver.cc:460] Webserver started at http://127.12.68.193:39019/ using document root <none> and password file <none>
I20260926 14:51:43.389349 23874 fs_manager.cc:359] Metadata directory not provided
I20260926 14:51:43.389411 23874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:51:43.389590 23874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:51:43 dist-test-slave-24mn krb5kdc[23317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:51:43.404430 23874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "84bb1813f2e841f08ce2e11ec5dda3da"
format_stamp: "Formatted at 2026-09-26 14:51:43 on dist-test-slave-24mn"
server_key: "22c22ecd61f82b0295ca23f07fe0b4e5"
server_key_iv: "4e500a3f13035a6ec68a2054e63f8e5e"
server_key_version: "kuduclusterkey@0"
I20260926 14:51:43.406777 23874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "84bb1813f2e841f08ce2e11ec5dda3da"
format_stamp: "Formatted at 2026-09-26 14:51:43 on dist-test-slave-24mn"
server_key: "22c22ecd61f82b0295ca23f07fe0b4e5"
server_key_iv: "4e500a3f13035a6ec68a2054e63f8e5e"
server_key_version: "kuduclusterkey@0"
I20260926 14:51:43.411242 23874 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260926 14:51:43.440388 23891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:43.442255 23874 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.001s sys 0.000s
I20260926 14:51:43.442380 23874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "84bb1813f2e841f08ce2e11ec5dda3da"
format_stamp: "Formatted at 2026-09-26 14:51:43 on dist-test-slave-24mn"
server_key: "22c22ecd61f82b0295ca23f07fe0b4e5"
server_key_iv: "4e500a3f13035a6ec68a2054e63f8e5e"
server_key_version: "kuduclusterkey@0"
I20260926 14:51:43.442523 23874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:51:43.497665 23874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:51:43.501248 23874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:51:43.501511 23874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:51:43.502244 23874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:51:43.503316 23874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:51:43.503422 23874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:43.503547 23874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:51:43.503644 23874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:43.514732 23874 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:36525
I20260926 14:51:43.514840 24006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:36525 every 8 connection(s)
Sep 26 14:51:43 dist-test-slave-24mn krb5kdc[23317](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:51:43.515959 23874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:51:43.525436 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 23874
I20260926 14:51:43.525589 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:51:43.525879 12563 external_mini_cluster.cc:1428] Setting key 08e804e74bd20128bfe009da55ca9ecf
Sep 26 14:51:43 dist-test-slave-24mn krb5kdc[23317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:51:43.535772 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:43.517944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:40917 (local address 127.12.68.254:44977)
0926 14:51:43.518200 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:43.518209 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:43.519173 (+ 964us) server_negotiation.cc:407] Connection header received
0926 14:51:43.520401 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:43.520404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:43.520460 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:43.520521 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:51:43.522150 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:43.522978 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:43.524643 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:43.524997 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:43.525110 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:43.531162 (+ 6052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:51:43.531165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:51:43.531168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:51:43.531195 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:51:43.533463 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:43.534057 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:43.534060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:43.534062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:43.534162 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:43.534525 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:43.534528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:43.534530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:43.534736 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:51:43.534810 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:43.535471 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 14:51:43.535594 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20260926 14:51:43.537513 24010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:43.518450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:44977 (local address 127.12.68.193:40917)
0926 14:51:43.519009 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:43.519064 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:43.520162 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:43.520999 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:43.521007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:43.521341 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:51:43.521972 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:43.521989 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:43.523137 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:43.523141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:43.524504 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:43.524515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:43.526627 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:43.526631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:43.526822 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:43.529293 (+ 2471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:51:43.529315 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:51:43.530960 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:51:43.533601 (+ 2641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:51:43.533607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:51:43.533626 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:51:43.533947 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:51:43.534260 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:51:43.534263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:51:43.534265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:51:43.534431 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:51:43.534874 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:51:43.534881 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:51:43.535204 (+ 323us) client_negotiation.cc:769] Sending connection context
0926 14:51:43.536167 (+ 963us) client_negotiation.cc:240] Negotiation successful
0926 14:51:43.536499 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":158,"threads_started":1}
I20260926 14:51:43.539062 24007 heartbeater.cc:347] Connected to a master server at 127.12.68.254:44977
I20260926 14:51:43.539289 24007 heartbeater.cc:464] Registering TS with master...
I20260926 14:51:43.539830 24007 heartbeater.cc:511] Master 127.12.68.254:44977 requested a full tablet report, sending...
I20260926 14:51:43.541502 23793 ts_manager.cc:194] Registered new tserver with Master: 84bb1813f2e841f08ce2e11ec5dda3da (127.12.68.193:36525)
I20260926 14:51:43.543035 23793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:40917
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:51:43.565014 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.68.212:56511/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:44977
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:38611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:51:43.694988 24013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:51:43.695232 24013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:51:43.695293 24013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:51:43.699199 24013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:51:43.699258 24013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:51:43.699332 24013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:51:43.704493 24013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:38611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.68.212:56511/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:44977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:51:43.705444 24013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:51:43.706382 24013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:51:43.715502 24022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:51:43.715509 24019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:43.715958 24013 server_base.cc:1034] running on GCE node
W20260926 14:51:43.716090 24020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:43.716442 24013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:51:43.717007 24013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:51:43.718181 24013 hybrid_clock.cc:648] HybridClock initialized: now 1790434303718163 us; error 30 us; skew 500 ppm
Sep 26 14:51:43 dist-test-slave-24mn krb5kdc[23317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:51:43.720795 24013 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:51:43.721750 24013 webserver.cc:460] Webserver started at http://127.12.68.194:40385/ using document root <none> and password file <none>
I20260926 14:51:43.722538 24013 fs_manager.cc:359] Metadata directory not provided
I20260926 14:51:43.722618 24013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:51:43.722848 24013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:51:43 dist-test-slave-24mn krb5kdc[23317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:51:43.736907 24013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "ed957c68420e444fb782d069dbe4ba99"
format_stamp: "Formatted at 2026-09-26 14:51:43 on dist-test-slave-24mn"
server_key: "68f51fb949ceea7d1e4f3df5f5960435"
server_key_iv: "b15726ed26a0910b4d9bf86ddf3fe955"
server_key_version: "kuduclusterkey@0"
I20260926 14:51:43.737874 24013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "ed957c68420e444fb782d069dbe4ba99"
format_stamp: "Formatted at 2026-09-26 14:51:43 on dist-test-slave-24mn"
server_key: "68f51fb949ceea7d1e4f3df5f5960435"
server_key_iv: "b15726ed26a0910b4d9bf86ddf3fe955"
server_key_version: "kuduclusterkey@0"
I20260926 14:51:43.753576 24013 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.004s sys 0.000s
I20260926 14:51:43.770884 24029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:43.772692 24013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:51:43.772825 24013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "ed957c68420e444fb782d069dbe4ba99"
format_stamp: "Formatted at 2026-09-26 14:51:43 on dist-test-slave-24mn"
server_key: "68f51fb949ceea7d1e4f3df5f5960435"
server_key_iv: "b15726ed26a0910b4d9bf86ddf3fe955"
server_key_version: "kuduclusterkey@0"
I20260926 14:51:43.772949 24013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:51:43.788529 24013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:51:43.792035 24013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:51:43.792248 24013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:51:43.792893 24013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:51:43.793973 24013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:51:43.794032 24013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:43.794082 24013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:51:43.794109 24013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:43.805406 24013 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:36525
I20260926 14:51:43.805449 24142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:36525 every 8 connection(s)
I20260926 14:51:43.806623 24013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:51:43.807168 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 24013
I20260926 14:51:43.807317 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:51:43.807650 12563 external_mini_cluster.cc:1428] Setting key 42df359363e4c057346517dfdfbc2e1f
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
Sep 26 14:51:43 dist-test-slave-24mn krb5kdc[23317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:51:43.839257 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:43.823590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:48489 (local address 127.12.68.254:44977)
0926 14:51:43.823719 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:43.823724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:43.824190 (+ 466us) server_negotiation.cc:407] Connection header received
0926 14:51:43.825404 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:43.825407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:43.825459 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:43.825510 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:51:43.827102 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:43.827928 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:43.829813 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:43.830163 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:43.830256 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:43.835309 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:51:43.835313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:51:43.835316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:51:43.835344 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:51:43.836909 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:43.837481 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:43.837485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:43.837487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:43.837572 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:43.837966 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:43.837969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:43.837971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:43.838197 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:51:43.838270 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:43.838948 (+ 678us) server_negotiation.cc:299] Negotiation successful
0926 14:51:43.839075 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:51:43.841077 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:43.822467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:44977 (local address 127.12.68.194:48489)
0926 14:51:43.824014 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:43.824077 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:43.825140 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:43.825919 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:43.825928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:43.826261 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:51:43.826931 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:43.826951 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:43.828081 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:43.828084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:43.829540 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:43.829551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:43.830557 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:43.830561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:43.830755 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:43.833413 (+ 2658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:51:43.833436 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:51:43.835097 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:51:43.837006 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:51:43.837012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:51:43.837028 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:51:43.837363 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:51:43.837683 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:51:43.837686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:51:43.837688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:51:43.837860 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:51:43.838350 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:51:43.838356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:51:43.838671 (+ 315us) client_negotiation.cc:769] Sending connection context
0926 14:51:43.839690 (+ 1019us) client_negotiation.cc:240] Negotiation successful
0926 14:51:43.840041 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1304,"thread_start_us":144,"threads_started":1}
I20260926 14:51:43.842577 24143 heartbeater.cc:347] Connected to a master server at 127.12.68.254:44977
I20260926 14:51:43.842832 24143 heartbeater.cc:464] Registering TS with master...
I20260926 14:51:43.843402 24143 heartbeater.cc:511] Master 127.12.68.254:44977 requested a full tablet report, sending...
I20260926 14:51:43.844515 23793 ts_manager.cc:194] Registered new tserver with Master: ed957c68420e444fb782d069dbe4ba99 (127.12.68.194:36525)
I20260926 14:51:43.845187 23793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:48489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:51:43.860800 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.68.212:56511/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:44977
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:38611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:51:43 dist-test-slave-24mn krb5kdc[23317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
W20260926 14:51:44.020076 24148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:51:44.020493 24148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:51:44.020586 24148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:51:44.026283 24148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:51:44.026350 24148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:51:44.026428 24148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:51:44.033885 24148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:38611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.68.212:56511/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:44977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:51:44.035179 24148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:51:44.036446 24148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:51:44.047624 24154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:51:44.047634 24156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:44.048069 24148 server_base.cc:1034] running on GCE node
W20260926 14:51:44.048249 24153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:51:44.048656 24148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:51:44.049386 24148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:51:44.050549 24148 hybrid_clock.cc:648] HybridClock initialized: now 1790434304050520 us; error 40 us; skew 500 ppm
Sep 26 14:51:44 dist-test-slave-24mn krb5kdc[23317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434304, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:51:44.053795 24148 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:51:44.054817 24148 webserver.cc:460] Webserver started at http://127.12.68.195:34021/ using document root <none> and password file <none>
I20260926 14:51:44.055310 24148 fs_manager.cc:359] Metadata directory not provided
I20260926 14:51:44.055362 24148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:51:44.055513 24148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:51:44 dist-test-slave-24mn krb5kdc[23317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434304, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:51:44.069875 24148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "d7806ee63b71463ba76b6f47f7db39c3"
format_stamp: "Formatted at 2026-09-26 14:51:44 on dist-test-slave-24mn"
server_key: "5a2e0d1f0c78260c286f3ecc64f984e4"
server_key_iv: "68cb4d2d1fe6af9b008366a36a0a85a6"
server_key_version: "kuduclusterkey@0"
I20260926 14:51:44.070688 24148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "d7806ee63b71463ba76b6f47f7db39c3"
format_stamp: "Formatted at 2026-09-26 14:51:44 on dist-test-slave-24mn"
server_key: "5a2e0d1f0c78260c286f3ecc64f984e4"
server_key_iv: "68cb4d2d1fe6af9b008366a36a0a85a6"
server_key_version: "kuduclusterkey@0"
I20260926 14:51:44.075676 24148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 14:51:44.089021 24163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:44.090746 24148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:51:44.090873 24148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "d7806ee63b71463ba76b6f47f7db39c3"
format_stamp: "Formatted at 2026-09-26 14:51:44 on dist-test-slave-24mn"
server_key: "5a2e0d1f0c78260c286f3ecc64f984e4"
server_key_iv: "68cb4d2d1fe6af9b008366a36a0a85a6"
server_key_version: "kuduclusterkey@0"
I20260926 14:51:44.091974 24148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:51:44.124992 24148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:51:44.128494 24148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:51:44.128680 24148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:51:44.129385 24148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:51:44.130491 24148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:51:44.130553 24148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:44.130609 24148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:51:44.130636 24148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:51:44.141390 24148 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:36733
I20260926 14:51:44.141431 24276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:36733 every 8 connection(s)
I20260926 14:51:44.142524 24148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:51:44.150292 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 24148
I20260926 14:51:44.150416 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:51:44.150696 12563 external_mini_cluster.cc:1428] Setting key 7004273526520c26024514e64ed3aece
Sep 26 14:51:44 dist-test-slave-24mn krb5kdc[23317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434304, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:51:44.159953 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:44.146088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:57769 (local address 127.12.68.254:44977)
0926 14:51:44.146208 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:44.146212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:44.146237 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:51:44.146846 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:44.146851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:44.146906 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:44.146994 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:51:44.148656 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.149511 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.152527 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.152865 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.152943 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:44.155971 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:51:44.155976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:51:44.155978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:51:44.156005 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:51:44.157648 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:44.158216 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:44.158219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:44.158221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:44.158318 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:44.158692 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:44.158696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:44.158697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:44.158900 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:51:44.158973 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:44.159616 (+ 643us) server_negotiation.cc:299] Negotiation successful
0926 14:51:44.159738 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:51:44.161017 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:44.145007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:44977 (local address 127.12.68.195:57769)
0926 14:51:44.145558 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:44.145613 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:44.146614 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:44.147313 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:44.147322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:44.147768 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:51:44.148486 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:44.148506 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.149742 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.149745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:44.152390 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:44.152401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.153208 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.153212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:44.153420 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:44.154087 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:51:44.154107 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:51:44.155766 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:51:44.157783 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:51:44.157789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:51:44.157805 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:51:44.158109 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:51:44.158422 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:51:44.158425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:51:44.158427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:51:44.158588 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:51:44.159032 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:51:44.159038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:51:44.159336 (+ 298us) client_negotiation.cc:769] Sending connection context
0926 14:51:44.159523 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 14:51:44.159840 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":165,"threads_started":1}
I20260926 14:51:44.162161 24277 heartbeater.cc:347] Connected to a master server at 127.12.68.254:44977
I20260926 14:51:44.162407 24277 heartbeater.cc:464] Registering TS with master...
I20260926 14:51:44.162940 24277 heartbeater.cc:511] Master 127.12.68.254:44977 requested a full tablet report, sending...
I20260926 14:51:44.164018 23792 ts_manager.cc:194] Registered new tserver with Master: d7806ee63b71463ba76b6f47f7db39c3 (127.12.68.195:36733)
I20260926 14:51:44.164700 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:51:44.164741 23792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:57769
I20260926 14:51:44.176319 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:44.167285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51280 (local address 127.12.68.254:44977)
0926 14:51:44.167389 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:44.167393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:44.167620 (+ 227us) server_negotiation.cc:407] Connection header received
0926 14:51:44.167785 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:44.167789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:44.167831 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:44.167877 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:51:44.168827 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.169631 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.170854 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.171205 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.171302 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:44.173007 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:51:44.173010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:51:44.173013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:51:44.173036 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:51:44.174595 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:44.175034 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:44.175037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:44.175039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:44.175114 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:51:44.175417 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:51:44.175421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:51:44.175422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:51:44.175609 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:51:44.175704 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:44.176053 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 14:51:44.176163 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:51:44.179868 23792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:51:44.184118 23792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:51:44.200721 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:44.193640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:36525 (local address 127.0.0.1:60816)
0926 14:51:44.194028 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:44.194080 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:44.194206 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:44.195371 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:44.195377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:44.195400 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:51:44.195532 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:44.195538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.197535 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.197538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:44.198872 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:44.198882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.200200 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.200204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:44.200324 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:44.200361 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 14:51:44.200462 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 14:51:44.200574 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
I20260926 14:51:44.201732 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:44.194447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.12.68.194:36525)
0926 14:51:44.194750 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:44.194761 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:44.194785 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:51:44.194856 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:44.194861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:44.195056 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:44.195211 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:51:44.195649 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.197413 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.199032 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.199871 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.199987 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:44.200131 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:44.201492 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0926 14:51:44.201592 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":99,"threads_started":1}
I20260926 14:51:44.203377 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:44.195949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:36525 (local address 127.0.0.1:42500)
0926 14:51:44.196593 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:44.196605 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:44.196686 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:44.197499 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:44.197502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:44.197520 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:51:44.197630 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:44.197637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.199718 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.199722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:44.201753 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:44.201763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.203001 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.203005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:44.203092 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:44.203114 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:51:44.203181 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:51:44.203245 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260926 14:51:44.203864 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:44.196079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42500 (local address 127.12.68.193:36525)
0926 14:51:44.196902 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:44.196908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:44.196929 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:51:44.197002 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:44.197007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:44.197197 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:44.197354 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:51:44.198488 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.199590 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.201888 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.202684 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.202804 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:44.202938 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:44.203621 (+ 683us) server_negotiation.cc:299] Negotiation successful
0926 14:51:44.203717 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":57,"threads_started":1}
I20260926 14:51:44.204798 24077 tablet_service.cc:1467] Processing CreateTablet for tablet 2c690cfa4ca247348e47fddc33313374 (DEFAULT_TABLE table=test-table [id=100c483502a242f58d223511a0efc09a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:51:44.205884 24077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c690cfa4ca247348e47fddc33313374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:51:44.206501 23941 tablet_service.cc:1467] Processing CreateTablet for tablet 2c690cfa4ca247348e47fddc33313374 (DEFAULT_TABLE table=test-table [id=100c483502a242f58d223511a0efc09a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:51:44.207511 23941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c690cfa4ca247348e47fddc33313374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:51:44.217716 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:44.196949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.12.68.195:36733)
0926 14:51:44.204518 (+ 7569us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:44.204529 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:44.204551 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:51:44.204626 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:44.204631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:44.205650 (+ 1019us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:44.205809 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:51:44.212194 (+ 6385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.213362 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.216116 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.216945 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.217022 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:44.217145 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:44.217476 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 14:51:44.217577 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7467,"thread_start_us":128,"threads_started":1}
I20260926 14:51:44.218058 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:44.196587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:36733 (local address 127.0.0.1:35252)
0926 14:51:44.201312 (+ 4725us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:44.201324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:44.201383 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:44.211947 (+ 10564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:44.211950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:44.211969 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:51:44.212047 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:44.212053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.213499 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.213503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:44.214636 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:44.214646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.217231 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.217246 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:44.217331 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:44.217352 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:51:44.217858 (+ 506us) client_negotiation.cc:240] Negotiation successful
0926 14:51:44.217934 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4671,"thread_start_us":74,"threads_started":1}
I20260926 14:51:44.220270 24296 tablet_bootstrap.cc:492] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99: Bootstrap starting.
I20260926 14:51:44.220924 24206 tablet_service.cc:1467] Processing CreateTablet for tablet 2c690cfa4ca247348e47fddc33313374 (DEFAULT_TABLE table=test-table [id=100c483502a242f58d223511a0efc09a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:51:44.222009 24206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c690cfa4ca247348e47fddc33313374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:51:44.222671 24295 tablet_bootstrap.cc:492] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da: Bootstrap starting.
I20260926 14:51:44.224237 24296 tablet_bootstrap.cc:654] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99: Neither blocks nor log segments found. Creating new log.
I20260926 14:51:44.227563 24295 tablet_bootstrap.cc:654] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da: Neither blocks nor log segments found. Creating new log.
I20260926 14:51:44.228114 24296 log.cc:824] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99: Log is configured to *not* fsync() on all Append() calls
I20260926 14:51:44.229372 24295 log.cc:824] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da: Log is configured to *not* fsync() on all Append() calls
I20260926 14:51:44.231408 24296 tablet_bootstrap.cc:492] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99: No bootstrap required, opened a new log
I20260926 14:51:44.231601 24296 ts_tablet_manager.cc:1397] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20260926 14:51:44.235412 24296 raft_consensus.cc:348] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed957c68420e444fb782d069dbe4ba99" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36525 } } peers { permanent_uuid: "84bb1813f2e841f08ce2e11ec5dda3da" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 36525 } } peers { permanent_uuid: "d7806ee63b71463ba76b6f47f7db39c3" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36733 } }
I20260926 14:51:44.235688 24296 raft_consensus.cc:374] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:51:44.235740 24296 raft_consensus.cc:729] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed957c68420e444fb782d069dbe4ba99, State: Initialized, Role: FOLLOWER
I20260926 14:51:44.236227 24296 consensus_queue.cc:260] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed957c68420e444fb782d069dbe4ba99" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36525 } } peers { permanent_uuid: "84bb1813f2e841f08ce2e11ec5dda3da" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 36525 } } peers { permanent_uuid: "d7806ee63b71463ba76b6f47f7db39c3" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36733 } }
I20260926 14:51:44.237212 24296 ts_tablet_manager.cc:1428] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260926 14:51:44.238016 24143 heartbeater.cc:503] Master 127.12.68.254:44977 was elected leader, sending a full tablet report...
I20260926 14:51:44.239560 24295 tablet_bootstrap.cc:492] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da: No bootstrap required, opened a new log
I20260926 14:51:44.239833 24295 ts_tablet_manager.cc:1397] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260926 14:51:44.243633 24299 tablet_bootstrap.cc:492] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3: Bootstrap starting.
I20260926 14:51:44.245364 24295 raft_consensus.cc:348] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed957c68420e444fb782d069dbe4ba99" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36525 } } peers { permanent_uuid: "84bb1813f2e841f08ce2e11ec5dda3da" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 36525 } } peers { permanent_uuid: "d7806ee63b71463ba76b6f47f7db39c3" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36733 } }
I20260926 14:51:44.245711 24295 raft_consensus.cc:374] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:51:44.245802 24295 raft_consensus.cc:729] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bb1813f2e841f08ce2e11ec5dda3da, State: Initialized, Role: FOLLOWER
I20260926 14:51:44.246356 24295 consensus_queue.cc:260] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed957c68420e444fb782d069dbe4ba99" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36525 } } peers { permanent_uuid: "84bb1813f2e841f08ce2e11ec5dda3da" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 36525 } } peers { permanent_uuid: "d7806ee63b71463ba76b6f47f7db39c3" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36733 } }
I20260926 14:51:44.246466 24299 tablet_bootstrap.cc:654] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3: Neither blocks nor log segments found. Creating new log.
I20260926 14:51:44.247294 24299 log.cc:824] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3: Log is configured to *not* fsync() on all Append() calls
I20260926 14:51:44.248618 24295 ts_tablet_manager.cc:1428] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da: Time spent starting tablet: real 0.009s user 0.001s sys 0.005s
I20260926 14:51:44.249013 24007 heartbeater.cc:503] Master 127.12.68.254:44977 was elected leader, sending a full tablet report...
I20260926 14:51:44.249392 24299 tablet_bootstrap.cc:492] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3: No bootstrap required, opened a new log
I20260926 14:51:44.249568 24299 ts_tablet_manager.cc:1397] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:51:44.253319 24299 raft_consensus.cc:348] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed957c68420e444fb782d069dbe4ba99" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36525 } } peers { permanent_uuid: "84bb1813f2e841f08ce2e11ec5dda3da" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 36525 } } peers { permanent_uuid: "d7806ee63b71463ba76b6f47f7db39c3" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36733 } }
I20260926 14:51:44.253582 24299 raft_consensus.cc:374] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:51:44.253638 24299 raft_consensus.cc:729] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7806ee63b71463ba76b6f47f7db39c3, State: Initialized, Role: FOLLOWER
I20260926 14:51:44.254112 24299 consensus_queue.cc:260] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed957c68420e444fb782d069dbe4ba99" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36525 } } peers { permanent_uuid: "84bb1813f2e841f08ce2e11ec5dda3da" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 36525 } } peers { permanent_uuid: "d7806ee63b71463ba76b6f47f7db39c3" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36733 } }
I20260926 14:51:44.255067 24299 ts_tablet_manager.cc:1428] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:51:44.255400 24277 heartbeater.cc:503] Master 127.12.68.254:44977 was elected leader, sending a full tablet report...
W20260926 14:51:44.267513 24008 tablet.cc:2369] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 14:51:44 dist-test-slave-24mn krb5kdc[23317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
W20260926 14:51:44.322139 24144 tablet.cc:2369] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:51:44.386917 24300 raft_consensus.cc:482] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:51:44.387135 24300 raft_consensus.cc:504] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed957c68420e444fb782d069dbe4ba99" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36525 } } peers { permanent_uuid: "84bb1813f2e841f08ce2e11ec5dda3da" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 36525 } } peers { permanent_uuid: "d7806ee63b71463ba76b6f47f7db39c3" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36733 } }
I20260926 14:51:44.388480 24300 leader_election.cc:288] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84bb1813f2e841f08ce2e11ec5dda3da (127.12.68.193:36525), d7806ee63b71463ba76b6f47f7db39c3 (127.12.68.195:36733)
I20260926 14:51:44.396456 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:44.389649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:45791 (local address 127.12.68.193:36525)
0926 14:51:44.389762 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:44.389767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:44.392116 (+ 2349us) server_negotiation.cc:407] Connection header received
0926 14:51:44.392295 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:44.392298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:44.392345 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:44.392389 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:51:44.392755 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.393518 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.395096 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.395730 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.395832 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:44.395909 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:44.396244 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 14:51:44.396319 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:51:44.396885 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:44.389494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:36525 (local address 127.12.68.194:45791)
0926 14:51:44.392035 (+ 2541us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:44.392048 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:44.392179 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:44.392510 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:44.392512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:44.392528 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:51:44.392631 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:44.392636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.393650 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.393653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:44.394947 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:44.394957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.396006 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.396009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:44.396118 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:44.396141 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:51:44.396655 (+ 514us) client_negotiation.cc:240] Negotiation successful
0926 14:51:44.396730 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2477,"thread_start_us":77,"threads_started":1}
I20260926 14:51:44.397656 23961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c690cfa4ca247348e47fddc33313374" candidate_uuid: "ed957c68420e444fb782d069dbe4ba99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bb1813f2e841f08ce2e11ec5dda3da" is_pre_election: true
I20260926 14:51:44.398038 23961 raft_consensus.cc:2454] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed957c68420e444fb782d069dbe4ba99 in term 0.
I20260926 14:51:44.398566 24033 leader_election.cc:302] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84bb1813f2e841f08ce2e11ec5dda3da, ed957c68420e444fb782d069dbe4ba99; no voters:
I20260926 14:51:44.398630 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:44.389218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:36943 (local address 127.12.68.195:36733)
0926 14:51:44.389360 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:44.389365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:44.390572 (+ 1207us) server_negotiation.cc:407] Connection header received
0926 14:51:44.390773 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:44.390776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:44.390822 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:44.390866 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:51:44.391295 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.392084 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.397324 (+ 5240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.397931 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.398029 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:44.398077 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:44.398442 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 14:51:44.398515 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:51:44.398875 24300 raft_consensus.cc:2781] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:51:44.398949 24300 raft_consensus.cc:482] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:51:44.398985 24300 raft_consensus.cc:3037] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:51:44.400318 24300 raft_consensus.cc:504] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed957c68420e444fb782d069dbe4ba99" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36525 } } peers { permanent_uuid: "84bb1813f2e841f08ce2e11ec5dda3da" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 36525 } } peers { permanent_uuid: "d7806ee63b71463ba76b6f47f7db39c3" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36733 } }
I20260926 14:51:44.400676 24300 leader_election.cc:288] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [CANDIDATE]: Term 1 election: Requested vote from peers 84bb1813f2e841f08ce2e11ec5dda3da (127.12.68.193:36525), d7806ee63b71463ba76b6f47f7db39c3 (127.12.68.195:36733)
I20260926 14:51:44.401592 23961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c690cfa4ca247348e47fddc33313374" candidate_uuid: "ed957c68420e444fb782d069dbe4ba99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bb1813f2e841f08ce2e11ec5dda3da"
I20260926 14:51:44.401715 23961 raft_consensus.cc:3037] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:51:44.402953 23961 raft_consensus.cc:2454] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed957c68420e444fb782d069dbe4ba99 in term 1.
I20260926 14:51:44.403334 24033 leader_election.cc:302] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84bb1813f2e841f08ce2e11ec5dda3da, ed957c68420e444fb782d069dbe4ba99; no voters:
I20260926 14:51:44.403551 24300 raft_consensus.cc:2781] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:51:44.403759 24300 raft_consensus.cc:686] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [term 1 LEADER]: Becoming Leader. State: Replica: ed957c68420e444fb782d069dbe4ba99, State: Running, Role: LEADER
I20260926 14:51:44.404145 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:44.388828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:36733 (local address 127.12.68.194:36943)
0926 14:51:44.390479 (+ 1651us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:44.390494 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:44.390649 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:44.391002 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:51:44.391005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:51:44.391022 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:51:44.391130 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:44.391136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.395962 (+ 4826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.395966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:44.397177 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:51:44.397189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.398206 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.398209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:51:44.398307 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:44.398329 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:51:44.398799 (+ 470us) client_negotiation.cc:240] Negotiation successful
0926 14:51:44.398871 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1579,"thread_start_us":80,"threads_started":1}
I20260926 14:51:44.404145 24300 consensus_queue.cc:237] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed957c68420e444fb782d069dbe4ba99" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36525 } } peers { permanent_uuid: "84bb1813f2e841f08ce2e11ec5dda3da" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 36525 } } peers { permanent_uuid: "d7806ee63b71463ba76b6f47f7db39c3" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36733 } }
I20260926 14:51:44.405032 24230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c690cfa4ca247348e47fddc33313374" candidate_uuid: "ed957c68420e444fb782d069dbe4ba99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7806ee63b71463ba76b6f47f7db39c3"
I20260926 14:51:44.405251 24230 raft_consensus.cc:3037] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:51:44.406560 24230 raft_consensus.cc:2454] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed957c68420e444fb782d069dbe4ba99 in term 1.
I20260926 14:51:44.406901 24231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c690cfa4ca247348e47fddc33313374" candidate_uuid: "ed957c68420e444fb782d069dbe4ba99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7806ee63b71463ba76b6f47f7db39c3" is_pre_election: true
I20260926 14:51:44.407018 24231 raft_consensus.cc:2362] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ed957c68420e444fb782d069dbe4ba99 in term 1. Re-sending same reply.
I20260926 14:51:44.410727 23792 catalog_manager.cc:5495] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed957c68420e444fb782d069dbe4ba99 (127.12.68.194). New cstate: current_term: 1 leader_uuid: "ed957c68420e444fb782d069dbe4ba99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed957c68420e444fb782d069dbe4ba99" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84bb1813f2e841f08ce2e11ec5dda3da" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 36525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7806ee63b71463ba76b6f47f7db39c3" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36733 } health_report { overall_health: UNKNOWN } } }
W20260926 14:51:44.418718 24278 tablet.cc:2369] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:51:44.447161 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:51:44.441314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60822 (local address 127.12.68.194:36525)
0926 14:51:44.441716 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 14:51:44.441721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:51:44.442006 (+ 285us) server_negotiation.cc:407] Connection header received
0926 14:51:44.442100 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:51:44.442103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:51:44.442173 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:51:44.442290 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:51:44.442658 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.443442 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.444898 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:51:44.445240 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:51:44.445350 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:51:44.445802 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:51:44.446398 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:51:44.446522 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 14:51:44.446956 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 14:51:44.447019 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306}
I20260926 14:51:44.457086 24231 raft_consensus.cc:1261] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3 [term 1 FOLLOWER]: Refusing update from remote peer ed957c68420e444fb782d069dbe4ba99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:51:44.458048 24308 consensus_queue.cc:1035] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7806ee63b71463ba76b6f47f7db39c3" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:51:44.465178 23961 raft_consensus.cc:1261] T 2c690cfa4ca247348e47fddc33313374 P 84bb1813f2e841f08ce2e11ec5dda3da [term 1 FOLLOWER]: Refusing update from remote peer ed957c68420e444fb782d069dbe4ba99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:51:44.465930 24308 consensus_queue.cc:1035] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84bb1813f2e841f08ce2e11ec5dda3da" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 36525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:51:44.479008 24311 mvcc.cc:205] Tried to move back new op lower bound from 7333618911048044544 to 7333618910856118272. Current Snapshot: MvccSnapshot[applied={T|T < 7333618911048044544}]
I20260926 14:51:44.493304 24314 mvcc.cc:205] Tried to move back new op lower bound from 7333618911048044544 to 7333618910856118272. Current Snapshot: MvccSnapshot[applied={T|T < 7333618911048044544}]
I20260926 14:51:44.502511 23792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:51:44.502707 23792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:51:44.505376 23792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e9f54a6c7cd49efbfc17dd77c989bca: Sending DeleteTablet for 3 replicas of tablet 2c690cfa4ca247348e47fddc33313374
I20260926 14:51:44.506594 24077 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c690cfa4ca247348e47fddc33313374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:51:44 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:60816
I20260926 14:51:44.507129 24206 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c690cfa4ca247348e47fddc33313374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:51:44 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:35252
I20260926 14:51:44.508117 23941 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c690cfa4ca247348e47fddc33313374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:51:44 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:42500
I20260926 14:51:44.509367 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 23874
I20260926 14:51:44.514889 24320 tablet_replica.cc:330] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99: stopping tablet replica
I20260926 14:51:44.515152 24321 tablet_replica.cc:330] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3: stopping tablet replica
I20260926 14:51:44.515563 24321 raft_consensus.cc:2229] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:51:44.515939 24320 raft_consensus.cc:2229] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:51:44.516436 24320 raft_consensus.cc:2258] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:51:44.516824 24321 raft_consensus.cc:2258] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:51:44.518620 24321 ts_tablet_manager.cc:1905] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:51:44.519944 24320 ts_tablet_manager.cc:1905] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:51:44.522805 24320 ts_tablet_manager.cc:1918] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:51:44.525537 24321 ts_tablet_manager.cc:1918] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 14:51:44.529181 23780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:36525 (error 108)
I20260926 14:51:44.529354 24321 log.cc:1196] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/2c690cfa4ca247348e47fddc33313374
I20260926 14:51:44.529543 24320 log.cc:1196] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/2c690cfa4ca247348e47fddc33313374
I20260926 14:51:44.529973 24321 ts_tablet_manager.cc:1939] T 2c690cfa4ca247348e47fddc33313374 P d7806ee63b71463ba76b6f47f7db39c3: Deleting consensus metadata
I20260926 14:51:44.530309 24320 ts_tablet_manager.cc:1939] T 2c690cfa4ca247348e47fddc33313374 P ed957c68420e444fb782d069dbe4ba99: Deleting consensus metadata
I20260926 14:51:44.531601 23777 catalog_manager.cc:4841] TS ed957c68420e444fb782d069dbe4ba99 (127.12.68.194:36525): tablet 2c690cfa4ca247348e47fddc33313374 (table test-table [id=100c483502a242f58d223511a0efc09a]) successfully deleted
I20260926 14:51:44.533013 23779 catalog_manager.cc:4841] TS d7806ee63b71463ba76b6f47f7db39c3 (127.12.68.195:36733): tablet 2c690cfa4ca247348e47fddc33313374 (table test-table [id=100c483502a242f58d223511a0efc09a]) successfully deleted
W20260926 14:51:44.536294 24289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:51:44.534626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:36525 (local address 127.0.0.1:42514)
0926 14:51:44.534822 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 14:51:44.534839 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:51:44.534927 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:51:44.536186 (+ 1259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:36525: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":107}
W20260926 14:51:44.536509 23780 catalog_manager.cc:4584] TS 84bb1813f2e841f08ce2e11ec5dda3da (127.12.68.193:36525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c690cfa4ca247348e47fddc33313374: Network error: Client connection negotiation failed: client connection to 127.12.68.193:36525: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:51:44.537892 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 24013
I20260926 14:51:44.543241 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 24148
I20260926 14:51:44.550076 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 23761
I20260926 14:51:44.557309 12563 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 14:51:44.891377 12563 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 14:51:44.891505 12563 mini_ranger.cc:66] Stopping Ranger...
I20260926 14:51:45.225445 12563 mini_ranger.cc:68] Stopped Ranger
2026-09-26 14:51:45.225 UTC [23347] LOG: received smart shutdown request
2026-09-26 14:51:45.229 UTC [23347] LOG: background worker "logical replication launcher" (PID 23356) exited with exit code 1
2026-09-26 14:51:45.229 UTC [23351] LOG: shutting down
2026-09-26 14:51:45.367 UTC [23347] LOG: database system is shut down
2026-09-26T14:51:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (42991 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:51:45 dist-test-slave-24mn krb5kdc[24332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:51:45 dist-test-slave-24mn krb5kdc[24332](info): set up 2 sockets
Sep 26 14:51:45 dist-test-slave-24mn krb5kdc[24332](info): commencing operation
krb5kdc: starting...
W20260926 14:51:46.411610 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:51:46 dist-test-slave-24mn krb5kdc[24332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:51:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:51:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/rangeradmin_127.12.68.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/rangeradmin_127.12.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/rangerlookup_127.12.68.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/rangerlookup_127.12.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/HTTP_127.12.68.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/HTTP_127.12.68.212@KRBTEST.COM.keytab.
I20260926 14:51:46.555964 12563 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwwFvmP/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/postgres -l logfile start
2026-09-26 14:51:48.764 UTC [24362] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 14:51:48.764 UTC [24362] LOG: listening on IPv4 address "127.12.68.212", port 41303
2026-09-26 14:51:48.767 UTC [24362] LOG: listening on Unix socket "/tmp/.s.PGSQL.41303"
2026-09-26 14:51:48.785 UTC [24365] LOG: database system was shut down at 2026-09-26 14:51:47 UTC
2026-09-26 14:51:48.788 UTC [24362] LOG: database system is ready to accept connections
I20260926 14:51:49.667493 12563 mini_postgres.cc:96] Postgres bound to 41303
2026-09-26 14:51:49.673 UTC [24373] FATAL: database "slave" does not exist
127.12.68.212:41303 - accepting connections
I20260926 14:51:49.673771 12563 mini_ranger.cc:161] Starting Ranger...
I20260926 14:51:49.687436 12563 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 14:51:50.080087 12563 mini_ranger.cc:87] Created ranger Postgres database
I20260926 14:51:50.080152 12563 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-admin
2026-09-26 14:51:50,344 [I] DB FLAVOR :POSTGRES
2026-09-26 14:51:50,345 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:51:50,345 [I] Checking connection..
2026-09-26 14:51:50,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 14:51:50,569 [I] Checking connection passed.
2026-09-26 14:51:50,569 [I] --------- Verifying version history table ---------
2026-09-26 14:51:50,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:51:50,820 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:51:50,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:51:51,066 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:51:51,066 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 14:51:51,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 14:51:51.307 UTC [24437] WARNING: there is no transaction in progress
2026-09-26 14:51:51,312 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 14:51:51,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:51:51,559 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 14:51:51,559 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:51:51,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 14:51:51,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 14:51:52,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-24mn.c.gcp-upstream.internal','N') ;"
2026-09-26 14:51:52,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:51:52,493 [I] Table x_portal_user does not exist in database ranger
2026-09-26 14:51:52,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:51:52,739 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 14:51:52,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:51:52,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:51:53,204 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 14:51:53,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 14:51:54.084 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.098 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.114 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.128 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.142 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.157 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.217 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.224 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.238 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.250 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.260 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.273 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.282 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.288 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.294 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54.300 UTC [24545] WARNING: there is no transaction in progress
2026-09-26 14:51:54,850 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 14:51:54,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 14:51:55,070 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 14:51:55,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:51:55,317 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 14:51:55,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:51:55,550 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 14:51:55,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:51:55,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:51:56,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-24mn.c.gcp-upstream.internal';"
2026-09-26 14:51:56,244 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 14:51:56,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 14:51:56,487 [I] DB_PATCHES have already been applied
I20260926 14:51:56.492998 12563 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwwFvmP/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 14:51:56.493058 12563 mini_ranger.cc:193] Using host: 127.12.68.212
I20260926 14:51:56.496152 12563 mini_ranger.cc:227] Ranger admin URL: http://127.12.68.212:54835
Sep 26, 2026 2:51:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:51:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:51:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:51:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:51:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:51:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:51:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 14:51:57.464838 12563 mini_ranger.cc:160] Time spent starting Ranger: real 7.791s user 0.000s sys 0.008s
Sep 26, 2026 2:52:16 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:52:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 2:52:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:52:16 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:52:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:52:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:52:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:52:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:52:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:52:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:52:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:52:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:52:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:52:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:52:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:52:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:52:16 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:52:17 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 14:52:18.360394 12563 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/rangerkms_127.12.68.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/rangerkms_127.12.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.68.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.68.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.68.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/HTTP_127.12.68.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.68.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/HTTP_127.12.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 14:52:18 dist-test-slave-24mn krb5kdc[24332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434338, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 14:52:18.441413 12563 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 14:52:18.449795 12563 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 14:52:19.181293 12563 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 14:52:19.181361 12563 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-kms
I20260926 14:52:19.181380 12563 mini_ranger_kms.cc:227] Using postgres at 127.12.68.212:41303
2026-09-26 14:52:19,336 [I] DB FLAVOR :POSTGRES
2026-09-26 14:52:19,336 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:52:19,336 [I] Checking connection
2026-09-26 14:52:19,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 14:52:19,586 [I] connection success
2026-09-26 14:52:19,587 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 14:52:19,587 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 14:52:19,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 14:52:19,809 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 14:52:19,809 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:52:19,809 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 14:52:19,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.68.212:41303/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 14:52:20,067 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 14:52:20.519733 12563 mini_ranger_kms.cc:310] Created kms service
I20260926 14:52:20.709517 12563 mini_ranger_kms.cc:326] Created kudu user
I20260926 14:52:20.753115 12563 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 14:52:20 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434338, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:52:21.144364 12563 mini_ranger_kms.cc:384] Added ranger policy
I20260926 14:52:21.144507 12563 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwwFvmP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwwFvmP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 14:52:21.144526 12563 mini_ranger_kms.cc:240] Using host: 127.12.68.212
I20260926 14:52:21.147904 12563 mini_ranger_kms.cc:276] Ranger KMS PID: 24725
I20260926 14:52:21.147979 12563 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.68.212:36835
Sep 26, 2026 2:52:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:52:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:52:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:52:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:52:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:52:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 14:52:23.178853 12563 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.737s user 0.002s sys 0.137s
I20260926 14:52:23.178961 12563 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 14:52:23.179019 12563 mini_ranger_kms.cc:401] 127.12.68.212:36835/kms/v1/keys
Sep 26 14:52:23 dist-test-slave-24mn krb5kdc[24332](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434343, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.68.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 14:52:23 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434343, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.68.212@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
Sep 26 14:52:24 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434343, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.68.212@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
Sep 26 14:52:26 dist-test-slave-24mn krb5kdc[24332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:52:26.254580 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:42773
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:42773
--ranger_config_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.68.212:36835/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:26.356851 24776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:26.357074 24776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:26.357115 24776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:26.361008 24776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:52:26.361058 24776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:26.361076 24776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:26.361092 24776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:52:26.361119 24776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:52:26.366323 24776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.68.212:36835/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:42773
--ranger_config_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:42773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:26.367236 24776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:26.368211 24776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:52:26.376274 24776 server_base.cc:1034] running on GCE node
W20260926 14:52:26.376385 24782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:26.376307 24781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:26.376278 24784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:26.377123 24776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:26.378154 24776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:26.379319 24776 hybrid_clock.cc:648] HybridClock initialized: now 1790434346379289 us; error 41 us; skew 500 ppm
Sep 26 14:52:26 dist-test-slave-24mn krb5kdc[24332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434346, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:26.381884 24776 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:52:26.382723 24776 webserver.cc:460] Webserver started at http://127.12.68.254:32945/ using document root <none> and password file <none>
I20260926 14:52:26.383205 24776 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:26.383255 24776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:26.383458 24776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:52:26 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434346, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:52:26.428661 24776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "dccc4d93b2194db0a54c93e4346c5844"
format_stamp: "Formatted at 2026-09-26 14:52:26 on dist-test-slave-24mn"
server_key: "ff629cd46e37bad672726edccf326195"
server_key_iv: "a15734046ef8aac74fe56b6c795914c8"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:26.429260 24776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "dccc4d93b2194db0a54c93e4346c5844"
format_stamp: "Formatted at 2026-09-26 14:52:26 on dist-test-slave-24mn"
server_key: "ff629cd46e37bad672726edccf326195"
server_key_iv: "a15734046ef8aac74fe56b6c795914c8"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:26.432873 24776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:52:26.446055 24791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:26.447319 24776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:52:26.447417 24776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "dccc4d93b2194db0a54c93e4346c5844"
format_stamp: "Formatted at 2026-09-26 14:52:26 on dist-test-slave-24mn"
server_key: "ff629cd46e37bad672726edccf326195"
server_key_iv: "a15734046ef8aac74fe56b6c795914c8"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:26.447528 24776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:26.471403 24776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:26.473943 24776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:26.474097 24776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:26.481113 24776 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:42773
I20260926 14:52:26.481118 24843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:42773 every 8 connection(s)
I20260926 14:52:26.482015 24776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:52:26.484670 24844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:26.489867 24844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844: Bootstrap starting.
I20260926 14:52:26.491250 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 24776
I20260926 14:52:26.491387 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:52:26.491667 12563 external_mini_cluster.cc:1428] Setting key d548b6fe441d90fc585844f6e5184bbf
I20260926 14:52:26.492399 24844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:26.493296 24844 log.cc:824] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:26.495322 24844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844: No bootstrap required, opened a new log
I20260926 14:52:26.499168 24844 raft_consensus.cc:348] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccc4d93b2194db0a54c93e4346c5844" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42773 } }
I20260926 14:52:26.499395 24844 raft_consensus.cc:374] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:26.499465 24844 raft_consensus.cc:729] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccc4d93b2194db0a54c93e4346c5844, State: Initialized, Role: FOLLOWER
I20260926 14:52:26.499938 24844 consensus_queue.cc:260] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccc4d93b2194db0a54c93e4346c5844" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42773 } }
I20260926 14:52:26.500075 24844 raft_consensus.cc:388] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:52:26.500150 24844 raft_consensus.cc:482] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:52:26.500247 24844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:26.501075 24844 raft_consensus.cc:504] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccc4d93b2194db0a54c93e4346c5844" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42773 } }
Sep 26 14:52:26 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:26.501428 24844 leader_election.cc:302] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dccc4d93b2194db0a54c93e4346c5844; no voters:
I20260926 14:52:26.501715 24844 leader_election.cc:288] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:52:26.501844 24849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:52:26.502102 24849 raft_consensus.cc:686] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 1 LEADER]: Becoming Leader. State: Replica: dccc4d93b2194db0a54c93e4346c5844, State: Running, Role: LEADER
I20260926 14:52:26.502450 24849 consensus_queue.cc:237] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccc4d93b2194db0a54c93e4346c5844" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42773 } }
I20260926 14:52:26.503032 24844 sys_catalog.cc:564] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:52:26.503778 24849 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dccc4d93b2194db0a54c93e4346c5844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccc4d93b2194db0a54c93e4346c5844" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42773 } } }
I20260926 14:52:26.503978 24849 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:26.504200 24849 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dccc4d93b2194db0a54c93e4346c5844. Latest consensus state: current_term: 1 leader_uuid: "dccc4d93b2194db0a54c93e4346c5844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccc4d93b2194db0a54c93e4346c5844" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42773 } } }
I20260926 14:52:26.504320 24849 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:26.504602 24858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:52:26.505149 24844 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 14:52:26.508044 24858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:52:26.508883 24847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:26.492840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.12.68.254:42773)
0926 14:52:26.493295 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:26.493315 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:26.493356 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:52:26.494037 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:26.494058 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:26.494451 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:26.494784 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:26.495616 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:26.497101 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:26.498313 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:26.498981 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:26.499127 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:26.502103 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:26.502109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:26.502124 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:26.502151 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:26.504069 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:26.504583 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:26.504588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:26.504595 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:26.504763 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:26.506308 (+ 1545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:26.506311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:26.506313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:26.506754 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:26.507493 (+ 739us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:26.507630 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 14:52:26.508007 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":117,"threads_started":1}
I20260926 14:52:26.517907 24858 catalog_manager.cc:1347] Generated new cluster ID: 5140c1a42efb4a77bfdea905a3fd29a4
I20260926 14:52:26.517973 24858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:52:26.532882 24858 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:52:26.534157 24858 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:52:26.550432 24858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844: Generated new TSK 0
I20260926 14:52:26.551288 24858 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:52:27.215564 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.68.212:36835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42773
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:27.344584 24889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:27.344838 24889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:27.344897 24889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:27.348901 24889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:27.348958 24889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:27.349037 24889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:52:27.354355 24889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.68.212:36835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:27.355381 24889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:27.356405 24889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:27.365303 24899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:27.365378 24889 server_base.cc:1034] running on GCE node
W20260926 14:52:27.365315 24897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:27.365587 24896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:27.366156 24889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:27.366882 24889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:27.368095 24889 hybrid_clock.cc:648] HybridClock initialized: now 1790434347368058 us; error 53 us; skew 500 ppm
Sep 26 14:52:27 dist-test-slave-24mn krb5kdc[24332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434347, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:27.371630 24889 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:52:27.372661 24889 webserver.cc:460] Webserver started at http://127.12.68.193:36673/ using document root <none> and password file <none>
I20260926 14:52:27.373368 24889 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:27.373476 24889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:27.373744 24889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:52:27 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434347, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:52:27.390549 24889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872"
format_stamp: "Formatted at 2026-09-26 14:52:27 on dist-test-slave-24mn"
server_key: "1e08c5bb124d128b4049c1972d42bc6d"
server_key_iv: "a7639e72dd609329e042bcab2a3614aa"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:27.391263 24889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872"
format_stamp: "Formatted at 2026-09-26 14:52:27 on dist-test-slave-24mn"
server_key: "1e08c5bb124d128b4049c1972d42bc6d"
server_key_iv: "a7639e72dd609329e042bcab2a3614aa"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:27.395807 24889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:52:27.413748 24906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:27.415558 24889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:52:27.415741 24889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872"
format_stamp: "Formatted at 2026-09-26 14:52:27 on dist-test-slave-24mn"
server_key: "1e08c5bb124d128b4049c1972d42bc6d"
server_key_iv: "a7639e72dd609329e042bcab2a3614aa"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:27.415977 24889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:27.453192 24889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:27.456172 24889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:27.456337 24889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:27.456881 24889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:27.457695 24889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:27.457744 24889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:27.457794 24889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:27.457846 24889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:27.467034 24889 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:33337
I20260926 14:52:27.467073 25021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:33337 every 8 connection(s)
I20260926 14:52:27.468068 24889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:52:27.476135 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 24889
I20260926 14:52:27.476254 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:52:27.476547 12563 external_mini_cluster.cc:1428] Setting key 3422ef91386738a16a63ebbd07689647
Sep 26 14:52:27 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434347, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:27.481168 25024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:27.470001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:44589 (local address 127.12.68.254:42773)
0926 14:52:27.470287 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:27.470296 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:27.470985 (+ 689us) server_negotiation.cc:407] Connection header received
0926 14:52:27.471939 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:27.471942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:27.471999 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:27.472075 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:27.473071 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:27.473602 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:27.474670 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:27.474905 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:27.474991 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:27.477304 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:27.477307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:27.477309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:27.477330 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:27.478982 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:27.479520 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:27.479523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:27.479525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:27.479606 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:27.479980 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:27.479984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:27.479985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:27.480175 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:27.480237 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:27.480875 (+ 638us) server_negotiation.cc:299] Negotiation successful
0926 14:52:27.481004 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":115,"threads_started":1}
I20260926 14:52:27.482442 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:27.470431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42773 (local address 127.12.68.193:44589)
0926 14:52:27.470845 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:27.470895 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:27.471709 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:27.472255 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:27.472262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:27.472498 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:27.472937 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:27.472950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:27.473712 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:27.473716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:27.474559 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:27.474567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:27.475032 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:27.475036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:27.475191 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:27.475671 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:27.475686 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:27.477128 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:27.479101 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:27.479106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:27.479121 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:27.479417 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:27.479699 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:27.479702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:27.479704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:27.479856 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:27.480314 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:27.480319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:27.480597 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 14:52:27.481463 (+ 866us) client_negotiation.cc:240] Negotiation successful
0926 14:52:27.481721 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
I20260926 14:52:27.485355 25022 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42773
I20260926 14:52:27.485585 25022 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:27.486092 25022 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
I20260926 14:52:27.487741 24808 ts_manager.cc:194] Registered new tserver with Master: 2d9e753e43bc43dc8fb2b3ef5a6f4872 (127.12.68.193:33337)
I20260926 14:52:27.489257 24808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:44589
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:52:27.519520 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.68.212:36835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42773
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:52:27 dist-test-slave-24mn krb5kdc[24332](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434347, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 14:52:27.671752 25028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:27.672048 25028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:27.672111 25028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:27.676815 25028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:27.676941 25028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:27.677063 25028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:52:27.684703 25028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.68.212:36835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:27.686152 25028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:27.687419 25028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:27.699873 25035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:27.699875 25034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:27.700752 25028 server_base.cc:1034] running on GCE node
W20260926 14:52:27.701052 25037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:27.701526 25028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:27.702263 25028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:27.703408 25028 hybrid_clock.cc:648] HybridClock initialized: now 1790434347703344 us; error 82 us; skew 500 ppm
Sep 26 14:52:27 dist-test-slave-24mn krb5kdc[24332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434347, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:27.706795 25028 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:52:27.707747 25028 webserver.cc:460] Webserver started at http://127.12.68.194:45241/ using document root <none> and password file <none>
I20260926 14:52:27.708419 25028 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:27.708488 25028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:27.708694 25028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:52:27 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434347, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:52:27.727193 25028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "73eff62ef88244d293e8d919ef7bc4e8"
format_stamp: "Formatted at 2026-09-26 14:52:27 on dist-test-slave-24mn"
server_key: "e0988b11b7ea0fe3b539c077f459ad8e"
server_key_iv: "60c52cdb23a0a4a11161599a8580a463"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:27.728272 25028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "73eff62ef88244d293e8d919ef7bc4e8"
format_stamp: "Formatted at 2026-09-26 14:52:27 on dist-test-slave-24mn"
server_key: "e0988b11b7ea0fe3b539c077f459ad8e"
server_key_iv: "60c52cdb23a0a4a11161599a8580a463"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:27.734695 25028 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260926 14:52:27.762717 25044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:27.764605 25028 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.002s sys 0.000s
I20260926 14:52:27.764747 25028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "73eff62ef88244d293e8d919ef7bc4e8"
format_stamp: "Formatted at 2026-09-26 14:52:27 on dist-test-slave-24mn"
server_key: "e0988b11b7ea0fe3b539c077f459ad8e"
server_key_iv: "60c52cdb23a0a4a11161599a8580a463"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:27.764904 25028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:27.797051 25028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:27.799863 25028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:27.800056 25028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:27.800645 25028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:27.801453 25028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:27.801498 25028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:27.801553 25028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:27.801580 25028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:27.811192 25028 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:45355
I20260926 14:52:27.811324 25157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:45355 every 8 connection(s)
I20260926 14:52:27.812134 25028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:52:27.820111 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 25028
I20260926 14:52:27.820250 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:52:27.820580 12563 external_mini_cluster.cc:1428] Setting key cab2a13b9dc025c99f13ea5dde7387a4
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
Sep 26 14:52:27 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434347, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:27.852634 25024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:27.813859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:53281 (local address 127.12.68.254:42773)
0926 14:52:27.814006 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:27.814010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:27.828253 (+ 14243us) server_negotiation.cc:407] Connection header received
0926 14:52:27.838268 (+ 10015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:27.838273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:27.838340 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:27.838406 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:27.842330 (+ 3924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:27.843154 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:27.844862 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:27.845244 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:27.845368 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:27.847980 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:27.847984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:27.847987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:27.848017 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:27.849725 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:27.850399 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:27.850403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:27.850405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:27.850507 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:27.851172 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:27.851175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:27.851177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:27.851400 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:27.851482 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:27.852312 (+ 830us) server_negotiation.cc:299] Negotiation successful
0926 14:52:27.852471 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 14:52:27.854326 25160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:27.814244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42773 (local address 127.12.68.194:53281)
0926 14:52:27.828051 (+ 13807us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:27.828134 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:27.837928 (+ 9794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:27.838898 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:27.838909 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:27.839269 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:27.842111 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:27.842134 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:27.843337 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:27.843341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:27.844722 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:27.844732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:27.845438 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:27.845442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:27.845627 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:27.846260 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:27.846277 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:27.847760 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:27.849897 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:27.849904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:27.849921 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:27.850257 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:27.850627 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:27.850630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:27.850632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:27.851058 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:27.851606 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:27.851612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:27.852012 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 14:52:27.852933 (+ 921us) client_negotiation.cc:240] Negotiation successful
0926 14:52:27.853272 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":99,"threads_started":1}
I20260926 14:52:27.855741 25158 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:52:27.856040 25158 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:27.856603 25158 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
I20260926 14:52:27.857007 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.68.212:36835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42773
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:52:27.857693 24808 ts_manager.cc:194] Registered new tserver with Master: 73eff62ef88244d293e8d919ef7bc4e8 (127.12.68.194:45355)
I20260926 14:52:27.858438 24808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:53281
Sep 26 14:52:27 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434347, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
W20260926 14:52:28.011106 25163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:28.011833 25163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:28.011976 25163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:28.017863 25163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:28.018011 25163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:28.018133 25163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:52:28.025573 25163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.68.212:36835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:28.026981 25163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:28.028316 25163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:28.040225 25169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:28.040727 25168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:28.040922 25171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:28.041985 25163 server_base.cc:1034] running on GCE node
I20260926 14:52:28.042536 25163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:28.043249 25163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:28.050618 25163 hybrid_clock.cc:648] HybridClock initialized: now 1790434348050547 us; error 86 us; skew 500 ppm
Sep 26 14:52:28 dist-test-slave-24mn krb5kdc[24332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:28.054291 25163 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:52:28.055445 25163 webserver.cc:460] Webserver started at http://127.12.68.195:36581/ using document root <none> and password file <none>
I20260926 14:52:28.056216 25163 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:28.056366 25163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:28.056643 25163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:52:28 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:52:28.074685 25163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "d8df6dc6da974461bb6810cdb2662c6a"
format_stamp: "Formatted at 2026-09-26 14:52:28 on dist-test-slave-24mn"
server_key: "4825636607964bd022b0f4ef5239cef9"
server_key_iv: "4f8b1425985d596b3605bf351ee0a10c"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:28.075708 25163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "d8df6dc6da974461bb6810cdb2662c6a"
format_stamp: "Formatted at 2026-09-26 14:52:28 on dist-test-slave-24mn"
server_key: "4825636607964bd022b0f4ef5239cef9"
server_key_iv: "4f8b1425985d596b3605bf351ee0a10c"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:28.087054 25163 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.001s
I20260926 14:52:28.108861 25178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:28.110611 25163 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260926 14:52:28.110750 25163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "d8df6dc6da974461bb6810cdb2662c6a"
format_stamp: "Formatted at 2026-09-26 14:52:28 on dist-test-slave-24mn"
server_key: "4825636607964bd022b0f4ef5239cef9"
server_key_iv: "4f8b1425985d596b3605bf351ee0a10c"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:28.110908 25163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:28.131664 25163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:28.135448 25163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:28.135641 25163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:28.136349 25163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:28.137378 25163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:28.137440 25163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:28.137496 25163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:28.137526 25163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:28.147692 25163 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:37717
I20260926 14:52:28.148830 25163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:52:28.150552 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 25163
I20260926 14:52:28.150663 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:52:28.150972 12563 external_mini_cluster.cc:1428] Setting key 620f494c2dbc61fa089adec57813e4d3
I20260926 14:52:28.153178 25291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:37717 every 8 connection(s)
Sep 26 14:52:28 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:28.170794 25024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:28.155735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:52277 (local address 127.12.68.254:42773)
0926 14:52:28.155853 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:28.155857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:28.157711 (+ 1854us) server_negotiation.cc:407] Connection header received
0926 14:52:28.158943 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:28.158947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:28.158998 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:28.159048 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:28.160746 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:28.161535 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:28.163172 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:28.163500 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:28.163576 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:28.166535 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:28.166540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:28.166542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:28.166568 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:28.168247 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:28.169035 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:28.169038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:28.169040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:28.169125 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:28.169524 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:28.169528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:28.169529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:28.169770 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:28.169857 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:28.170482 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 14:52:28.170608 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:52:28.172643 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:28.156328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42773 (local address 127.12.68.195:52277)
0926 14:52:28.157558 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:28.157615 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:28.158682 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:28.159509 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:28.159519 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:28.159922 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:28.160581 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:28.160601 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:28.161690 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:28.161693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:28.163036 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:28.163047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:28.163764 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:28.163768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:28.163991 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:28.164648 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:28.164668 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:28.166355 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:28.168385 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:28.168391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:28.168406 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:28.168919 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:28.169234 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:28.169237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:28.169239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:28.169421 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:28.169921 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:28.169928 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:28.170219 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 14:52:28.171231 (+ 1012us) client_negotiation.cc:240] Negotiation successful
0926 14:52:28.171557 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"thread_start_us":156,"threads_started":1}
I20260926 14:52:28.174134 25292 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42773
I20260926 14:52:28.174358 25292 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:28.174906 25292 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
I20260926 14:52:28.175987 24808 ts_manager.cc:194] Registered new tserver with Master: d8df6dc6da974461bb6810cdb2662c6a (127.12.68.195:37717)
I20260926 14:52:28.176708 24808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:52277
I20260926 14:52:28.189018 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:52:28.189878 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 24889
I20260926 14:52:28.198604 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 25028
I20260926 14:52:28.206616 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 25163
I20260926 14:52:28.213166 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 24776
I20260926 14:52:28.218120 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:42773
--webserver_interface=127.12.68.254
--webserver_port=32945
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:42773
--ranger_config_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.68.212:36835/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.68.212:36835/kms with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:28.323472 25295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:28.323711 25295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:28.323748 25295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:28.327651 25295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:52:28.327702 25295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:28.327719 25295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:28.327733 25295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:52:28.327746 25295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:52:28.332960 25295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.68.212:36835/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:42773
--ranger_config_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:42773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=32945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:28.333843 25295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:28.334719 25295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:28.342614 25303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:28.342648 25301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:28.342643 25300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:28.342937 25295 server_base.cc:1034] running on GCE node
I20260926 14:52:28.343640 25295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:28.344673 25295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:28.345825 25295 hybrid_clock.cc:648] HybridClock initialized: now 1790434348345806 us; error 30 us; skew 500 ppm
Sep 26 14:52:28 dist-test-slave-24mn krb5kdc[24332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:28.348327 25295 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:52:28.349155 25295 webserver.cc:460] Webserver started at http://127.12.68.254:32945/ using document root <none> and password file <none>
I20260926 14:52:28.349648 25295 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:28.349728 25295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:52:28 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:52:28.392289 25295 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:52:28.394520 25311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:28.395917 25295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:52:28.396037 25295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "dccc4d93b2194db0a54c93e4346c5844"
format_stamp: "Formatted at 2026-09-26 14:52:26 on dist-test-slave-24mn"
server_key: "ff629cd46e37bad672726edccf326195"
server_key_iv: "a15734046ef8aac74fe56b6c795914c8"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:28.396598 25295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:28.411861 25295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:28.414500 25295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:28.414650 25295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:28.421262 25295 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:42773
I20260926 14:52:28.421285 25363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:42773 every 8 connection(s)
I20260926 14:52:28.422230 25295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:52:28.424094 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 25295
I20260926 14:52:28.424484 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.68.212:36835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:33337
--local_ip_for_outbound_sockets=127.12.68.193
--tserver_master_addrs=127.12.68.254:42773
--webserver_port=36673
--webserver_interface=127.12.68.193
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.68.212:36835/kms with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:52:28.426353 25364 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:52:28.428069 25364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844: Bootstrap starting.
I20260926 14:52:28.443799 25364 log.cc:824] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:28.449959 25364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:52:28.450295 25364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844: Bootstrap complete.
I20260926 14:52:28.454077 25364 raft_consensus.cc:348] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccc4d93b2194db0a54c93e4346c5844" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42773 } }
I20260926 14:52:28.454454 25364 raft_consensus.cc:729] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccc4d93b2194db0a54c93e4346c5844, State: Initialized, Role: FOLLOWER
I20260926 14:52:28.454897 25364 consensus_queue.cc:260] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccc4d93b2194db0a54c93e4346c5844" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42773 } }
I20260926 14:52:28.455013 25364 raft_consensus.cc:388] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:52:28.455077 25364 raft_consensus.cc:482] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:52:28.455165 25364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:52:28.456972 25364 raft_consensus.cc:504] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccc4d93b2194db0a54c93e4346c5844" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42773 } }
I20260926 14:52:28.457264 25364 leader_election.cc:302] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dccc4d93b2194db0a54c93e4346c5844; no voters:
I20260926 14:52:28.457535 25364 leader_election.cc:288] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:52:28.457631 25368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:52:28.457813 25368 raft_consensus.cc:686] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [term 2 LEADER]: Becoming Leader. State: Replica: dccc4d93b2194db0a54c93e4346c5844, State: Running, Role: LEADER
I20260926 14:52:28.458101 25368 consensus_queue.cc:237] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccc4d93b2194db0a54c93e4346c5844" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42773 } }
I20260926 14:52:28.458730 25364 sys_catalog.cc:564] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:52:28.458957 25369 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dccc4d93b2194db0a54c93e4346c5844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccc4d93b2194db0a54c93e4346c5844" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42773 } } }
I20260926 14:52:28.459074 25369 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:28.459734 25370 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dccc4d93b2194db0a54c93e4346c5844. Latest consensus state: current_term: 2 leader_uuid: "dccc4d93b2194db0a54c93e4346c5844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccc4d93b2194db0a54c93e4346c5844" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42773 } } }
I20260926 14:52:28.459921 25370 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844 [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:28.460304 25372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:52:28.463187 25372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:52:28.463697 25364 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 14:52:28.463801 25372 catalog_manager.cc:1259] Loaded cluster ID: 5140c1a42efb4a77bfdea905a3fd29a4
I20260926 14:52:28.463873 25372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:52:28.466199 25372 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:52:28.466838 25372 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844: Loaded TSK: 0
I20260926 14:52:28.468040 25372 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:52:28.552951 25366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:28.553177 25366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:28.553232 25366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:28.557307 25366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:28.557372 25366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:28.557451 25366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:52:28.563366 25366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.68.212:36835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:33337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=36673
--tserver_master_addrs=127.12.68.254:42773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:28.564375 25366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:28.565327 25366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:28.575222 25402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:28.575433 25404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:28.576033 25401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:28.576452 25366 server_base.cc:1034] running on GCE node
I20260926 14:52:28.576870 25366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:28.577555 25366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:28.578732 25366 hybrid_clock.cc:648] HybridClock initialized: now 1790434348578707 us; error 40 us; skew 500 ppm
Sep 26 14:52:28 dist-test-slave-24mn krb5kdc[24332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:28.581974 25366 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:52:28.583024 25366 webserver.cc:460] Webserver started at http://127.12.68.193:36673/ using document root <none> and password file <none>
I20260926 14:52:28.583698 25366 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:28.583802 25366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:52:28 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:52:28.606161 25366 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.000s
I20260926 14:52:28.609064 25412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:28.610605 25366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:52:28.610723 25366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872"
format_stamp: "Formatted at 2026-09-26 14:52:27 on dist-test-slave-24mn"
server_key: "1e08c5bb124d128b4049c1972d42bc6d"
server_key_iv: "a7639e72dd609329e042bcab2a3614aa"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:28.611354 25366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:28.635646 25366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:28.638665 25366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:28.638859 25366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:28.639449 25366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:28.640331 25366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:28.640383 25366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:28.640439 25366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:28.640461 25366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:28.652937 25366 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:33337
I20260926 14:52:28.653918 25366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:52:28.654881 25525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:33337 every 8 connection(s)
I20260926 14:52:28.658563 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 25366
I20260926 14:52:28.659026 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.68.212:36835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:45355
--local_ip_for_outbound_sockets=127.12.68.194
--tserver_master_addrs=127.12.68.254:42773
--webserver_port=45241
--webserver_interface=127.12.68.194
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.68.212:36835/kms with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:52:28 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:28.674343 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:28.656375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42773 (local address 127.12.68.193:45137)
0926 14:52:28.657157 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:28.657209 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:28.658015 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:28.659717 (+ 1702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:28.659732 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:28.660073 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:28.660707 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:28.660725 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:28.662489 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:28.662493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:28.663799 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:28.663811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:28.665067 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:28.665071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:28.665268 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:28.665929 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:28.665951 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:28.668568 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:28.671015 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:28.671021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:28.671035 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:28.671348 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:28.671717 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:28.671720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:28.671722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:28.671900 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:28.672619 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:28.672625 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:28.672898 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 14:52:28.673070 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 14:52:28.673385 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":104,"threads_started":1}
I20260926 14:52:28.676090 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:28.656247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:45137 (local address 127.12.68.254:42773)
0926 14:52:28.656887 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:28.656909 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:28.657335 (+ 426us) server_negotiation.cc:407] Connection header received
0926 14:52:28.658465 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:28.658480 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:28.658884 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:28.659228 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:28.660894 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:28.662337 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:28.663942 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:28.664673 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:28.664808 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:28.668803 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:28.668810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:28.668827 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:28.668861 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:28.670881 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:28.671485 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:28.671491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:28.671496 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:28.671624 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:28.671998 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:28.672002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:28.672004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:28.672456 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:28.672570 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:28.674944 (+ 2374us) server_negotiation.cc:299] Negotiation successful
0926 14:52:28.675212 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":126,"threads_started":1}
I20260926 14:52:28.678407 25526 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42773
I20260926 14:52:28.678632 25526 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:28.679137 25526 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
W20260926 14:52:28.681160 25526 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:42773 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 14:52:28.681270 25526 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:28.681424 25526 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
I20260926 14:52:28.682193 25526 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:28.682351 25526 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
W20260926 14:52:28.683073 25526 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:52:28.798489 25530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:28.798815 25530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:28.798879 25530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:28.803656 25530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:28.803712 25530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:28.803789 25530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:52:28.809430 25530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.68.212:36835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:45355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=45241
--tserver_master_addrs=127.12.68.254:42773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:28.810577 25530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:28.811627 25530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:28.822502 25536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:28.822542 25538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:28.822899 25535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:28.823633 25530 server_base.cc:1034] running on GCE node
I20260926 14:52:28.824093 25530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:28.824851 25530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:28.826047 25530 hybrid_clock.cc:648] HybridClock initialized: now 1790434348826025 us; error 33 us; skew 500 ppm
Sep 26 14:52:28 dist-test-slave-24mn krb5kdc[24332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:28.829843 25530 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:52:28.830883 25530 webserver.cc:460] Webserver started at http://127.12.68.194:45241/ using document root <none> and password file <none>
I20260926 14:52:28.831552 25530 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:28.831652 25530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:52:28 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:52:28.851063 25530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:52:28.853654 25546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:28.855006 25530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 14:52:28.855120 25530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "73eff62ef88244d293e8d919ef7bc4e8"
format_stamp: "Formatted at 2026-09-26 14:52:27 on dist-test-slave-24mn"
server_key: "e0988b11b7ea0fe3b539c077f459ad8e"
server_key_iv: "60c52cdb23a0a4a11161599a8580a463"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:28.855718 25530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:28.874701 25530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:28.878012 25530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:28.878199 25530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:28.878897 25530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:28.880002 25530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:28.880065 25530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:28.880120 25530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:28.880148 25530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:28.890938 25530 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:45355
I20260926 14:52:28.893473 25659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:45355 every 8 connection(s)
I20260926 14:52:28.894158 25530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:52:28.898290 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 25530
I20260926 14:52:28.898754 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.68.212:36835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:37717
--local_ip_for_outbound_sockets=127.12.68.195
--tserver_master_addrs=127.12.68.254:42773
--webserver_port=36581
--webserver_interface=127.12.68.195
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.68.212:36835/kms with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:52:28 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:28.917353 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:28.897633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:49171 (local address 127.12.68.254:42773)
0926 14:52:28.897747 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:28.897752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:28.897768 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:52:28.898387 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:28.898391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:28.898451 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:28.898503 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:28.900126 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:28.900980 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:28.902682 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:28.903024 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:28.903085 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:28.912359 (+ 9274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:28.912363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:28.912366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:28.912395 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:28.914031 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:28.915539 (+ 1508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:28.915542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:28.915544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:28.915634 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:28.916067 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:28.916071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:28.916073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:28.916285 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:28.916357 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:28.917051 (+ 694us) server_negotiation.cc:299] Negotiation successful
0926 14:52:28.917183 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:52:28.919092 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:28.896678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42773 (local address 127.12.68.194:49171)
0926 14:52:28.897115 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:28.897165 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:28.898134 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:28.898915 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:28.898925 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:28.899233 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:28.899958 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:28.899978 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:28.901206 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:28.901209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:28.902538 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:28.902550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:28.903242 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:28.903244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:28.903376 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:28.910378 (+ 7002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:28.910400 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:28.912147 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:28.915055 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:28.915061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:28.915078 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:28.915418 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:28.915747 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:28.915750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:28.915752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:28.915957 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:28.916439 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:28.916445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:28.916756 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 14:52:28.917732 (+ 976us) client_negotiation.cc:240] Negotiation successful
0926 14:52:28.918079 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":96,"threads_started":1}
I20260926 14:52:28.920643 25660 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42773
I20260926 14:52:28.920903 25660 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:28.921469 25660 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
W20260926 14:52:28.923110 25660 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:42773 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 14:52:28.923230 25660 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:28.923388 25660 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
I20260926 14:52:28.924197 25660 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:28.924364 25660 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
W20260926 14:52:28.925120 25660 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:52:29.037441 25663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:29.037685 25663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:29.037738 25663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:29.041908 25663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:29.041970 25663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:29.042048 25663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:52:29.048903 25663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.68.212:36835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:37717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=36581
--tserver_master_addrs=127.12.68.254:42773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:29.050041 25663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:29.051019 25663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:29.061220 25669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:29.061220 25670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:29.061475 25672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:29.062307 25663 server_base.cc:1034] running on GCE node
I20260926 14:52:29.062667 25663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:29.063290 25663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:29.066625 25663 hybrid_clock.cc:648] HybridClock initialized: now 1790434349066551 us; error 89 us; skew 500 ppm
Sep 26 14:52:29 dist-test-slave-24mn krb5kdc[24332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434349, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:29.070115 25663 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:52:29.071205 25663 webserver.cc:460] Webserver started at http://127.12.68.195:36581/ using document root <none> and password file <none>
I20260926 14:52:29.071987 25663 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:29.072137 25663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:52:29 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434349, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:52:29.097283 25663 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260926 14:52:29.100418 25680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:29.101878 25663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:52:29.102005 25663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "d8df6dc6da974461bb6810cdb2662c6a"
format_stamp: "Formatted at 2026-09-26 14:52:28 on dist-test-slave-24mn"
server_key: "4825636607964bd022b0f4ef5239cef9"
server_key_iv: "4f8b1425985d596b3605bf351ee0a10c"
server_key_version: "kuduclusterkey@0"
I20260926 14:52:29.102689 25663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:29.126577 25663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:29.129719 25663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:29.129889 25663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:29.130488 25663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:29.131487 25663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:29.131546 25663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:29.131604 25663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:29.131639 25663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:29.142241 25663 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:37717
I20260926 14:52:29.143435 25663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:52:29.146703 25793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:37717 every 8 connection(s)
I20260926 14:52:29.148631 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 25663
Sep 26 14:52:29 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434349, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:29.163086 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:29.151104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:48453 (local address 127.12.68.254:42773)
0926 14:52:29.151225 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:29.151230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:29.151248 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:52:29.151865 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:29.151869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:29.151941 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:29.151993 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:29.153626 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:29.154397 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:29.156123 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:29.156339 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:29.156423 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:29.159448 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:29.159451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:29.159453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:29.159476 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:29.161238 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:29.161723 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:29.161727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:29.161729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:29.161800 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:29.162092 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:29.162095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:29.162097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:29.162262 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:29.162344 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:29.162812 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 14:52:29.162927 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:52:29.164009 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:29.150065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42773 (local address 127.12.68.195:48453)
0926 14:52:29.150563 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:29.150604 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:29.151631 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:29.152428 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:29.152438 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:29.152797 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:29.153453 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:29.153475 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:29.154551 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:29.154555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:29.156010 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:29.156020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:29.156473 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:29.156477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:29.156631 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:29.157855 (+ 1224us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:29.157875 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:29.159296 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:29.161363 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:29.161369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:29.161385 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:29.161620 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:29.161878 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:29.161881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:29.161883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:29.162008 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:29.162357 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:29.162363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:29.162587 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 14:52:29.162752 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 14:52:29.163049 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":132,"threads_started":1}
I20260926 14:52:29.165048 25794 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42773
I20260926 14:52:29.165273 25794 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:29.165730 25794 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
W20260926 14:52:29.167098 25794 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:42773 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 14:52:29.167208 25794 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:29.167384 25794 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
I20260926 14:52:29.168097 25794 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:29.168255 25794 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
W20260926 14:52:29.168922 25794 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:52:29.174440 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:29.151466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.12.68.254:42773)
0926 14:52:29.151689 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:29.151697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:29.151711 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:52:29.151798 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:29.151801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:29.151852 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:29.151969 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:29.155961 (+ 3992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:29.156782 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:29.159049 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:29.159413 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:29.159512 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:29.162843 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:29.162847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:29.162850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:29.162875 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:29.172641 (+ 9766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:29.173157 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:29.173161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:29.173163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:29.173259 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:29.173495 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:29.173499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:29.173500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:29.173735 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:29.173928 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:29.174139 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 14:52:29.174266 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"spinlock_wait_cycles":6912,"thread_start_us":86,"threads_started":1}
Sep 26 14:52:29 dist-test-slave-24mn krb5kdc[24332](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434349, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:29.683288 25526 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:29.683583 25526 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
I20260926 14:52:29.684795 25328 ts_manager.cc:194] Registered new tserver with Master: 2d9e753e43bc43dc8fb2b3ef5a6f4872 (127.12.68.193:33337)
I20260926 14:52:29.686564 25328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:45137
Sep 26 14:52:29 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434349, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:52:29.925335 25660 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:29.925619 25660 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
I20260926 14:52:29.926277 25328 ts_manager.cc:194] Registered new tserver with Master: 73eff62ef88244d293e8d919ef7bc4e8 (127.12.68.194:45355)
I20260926 14:52:29.926985 25328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:49171
Sep 26 14:52:30 dist-test-slave-24mn krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434349, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for HTTP/127.12.68.212@KRBTEST.COM
I20260926 14:52:30.169138 25794 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:30.169413 25794 heartbeater.cc:511] Master 127.12.68.254:42773 requested a full tablet report, sending...
I20260926 14:52:30.170251 25328 ts_manager.cc:194] Registered new tserver with Master: d8df6dc6da974461bb6810cdb2662c6a (127.12.68.195:37717)
I20260926 14:52:30.170953 25328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:48453
I20260926 14:52:30.174057 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:52:30.189051 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.175478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.12.68.254:42773)
0926 14:52:30.176420 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:30.176424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:30.176434 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:52:30.176465 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:30.176467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:30.176503 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:30.176576 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:30.177251 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.177885 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.178715 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.178960 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.179046 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.185766 (+ 6720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:30.185771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:30.185773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:30.185803 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:30.187523 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:30.187932 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:30.187936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:30.187938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:30.188012 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:30.188253 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:30.188256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:30.188258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:30.188429 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:30.188543 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:30.188759 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 14:52:30.188875 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":73,"threads_started":1}
I20260926 14:52:30.192572 25328 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:52:30.196591 25328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:52:30.222309 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.209210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:33337 (local address 127.0.0.1:44366)
0926 14:52:30.210841 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:30.210890 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:30.211010 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:30.216604 (+ 5594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:30.216611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:30.216638 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:30.216844 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:30.216853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.218304 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.218308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:30.220411 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:30.220422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.221752 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.221756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:30.221867 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.221910 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 14:52:30.222019 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 14:52:30.222147 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1443,"thread_start_us":105,"threads_started":1}
I20260926 14:52:30.223635 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.209344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.12.68.193:33337)
0926 14:52:30.216078 (+ 6734us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:30.216085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:30.216109 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:52:30.216186 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:30.216192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:30.216420 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:30.216583 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:30.216988 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.218171 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.220553 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.221412 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.221516 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.221672 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:30.223392 (+ 1720us) server_negotiation.cc:299] Negotiation successful
0926 14:52:30.223497 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6628,"thread_start_us":100,"threads_started":1}
I20260926 14:52:30.226835 25459 tablet_service.cc:1467] Processing CreateTablet for tablet 1bcae952fa404a60941f95c806c3f73a (DEFAULT_TABLE table=test-table [id=fe3ef739dc4341c2ad46b863c7fc52c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:52:30.228073 25459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bcae952fa404a60941f95c806c3f73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:30.228595 25827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.210028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:45355 (local address 127.0.0.1:36528)
0926 14:52:30.211220 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:30.211231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:30.211298 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:30.224539 (+ 13241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:30.224543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:30.224561 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:30.224657 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:30.224663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.226085 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.226088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:30.227180 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:30.227191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.228216 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.228219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:30.228314 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.228340 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:52:30.228397 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:52:30.228462 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260926 14:52:30.228910 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.210304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:37717 (local address 127.0.0.1:45496)
0926 14:52:30.211438 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:30.211448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:30.211503 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:30.217428 (+ 5925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:30.217431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:30.217449 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:30.217550 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:30.217556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.225255 (+ 7699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.225258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:30.226425 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:30.226436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.228500 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.228503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:30.228604 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.228630 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:52:30.228699 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:52:30.228779 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1086,"mutex_wait_us":21,"thread_start_us":80,"threads_started":1}
I20260926 14:52:30.229250 25826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.210180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.12.68.194:45355)
0926 14:52:30.224083 (+ 13903us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:30.224090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:30.224113 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:52:30.224183 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:30.224188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:30.224386 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:30.224806 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:30.224939 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.225960 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.227332 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.228090 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.228848 (+ 758us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.228984 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:30.229036 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 14:52:30.229120 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13794,"thread_start_us":96,"threads_started":1}
I20260926 14:52:30.229411 25828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.212158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.12.68.195:37717)
0926 14:52:30.216857 (+ 4699us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:30.216864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:30.216886 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:52:30.216957 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:30.216962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:30.217151 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:30.217284 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:30.223975 (+ 6691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.225113 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.226566 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.228199 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.228300 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.228435 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:30.229176 (+ 741us) server_negotiation.cc:299] Negotiation successful
0926 14:52:30.229275 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4596,"thread_start_us":88,"threads_started":1}
I20260926 14:52:30.232007 25578 tablet_service.cc:1467] Processing CreateTablet for tablet 1bcae952fa404a60941f95c806c3f73a (DEFAULT_TABLE table=test-table [id=fe3ef739dc4341c2ad46b863c7fc52c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:52:30.232010 25719 tablet_service.cc:1467] Processing CreateTablet for tablet 1bcae952fa404a60941f95c806c3f73a (DEFAULT_TABLE table=test-table [id=fe3ef739dc4341c2ad46b863c7fc52c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:52:30.233011 25719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bcae952fa404a60941f95c806c3f73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:30.233011 25578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bcae952fa404a60941f95c806c3f73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:30.241989 25829 tablet_bootstrap.cc:492] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a: Bootstrap starting.
I20260926 14:52:30.244300 25829 tablet_bootstrap.cc:654] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:30.245168 25829 log.cc:824] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:30.247188 25831 tablet_bootstrap.cc:492] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8: Bootstrap starting.
I20260926 14:52:30.249311 25831 tablet_bootstrap.cc:654] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:30.250164 25831 log.cc:824] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:30.252535 25831 tablet_bootstrap.cc:492] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8: No bootstrap required, opened a new log
I20260926 14:52:30.252722 25831 ts_tablet_manager.cc:1397] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260926 14:52:30.256503 25831 raft_consensus.cc:348] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73eff62ef88244d293e8d919ef7bc4e8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45355 } } peers { permanent_uuid: "d8df6dc6da974461bb6810cdb2662c6a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 37717 } } peers { permanent_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33337 } }
I20260926 14:52:30.256879 25831 raft_consensus.cc:374] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:30.256968 25831 raft_consensus.cc:729] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73eff62ef88244d293e8d919ef7bc4e8, State: Initialized, Role: FOLLOWER
I20260926 14:52:30.257501 25831 consensus_queue.cc:260] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73eff62ef88244d293e8d919ef7bc4e8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45355 } } peers { permanent_uuid: "d8df6dc6da974461bb6810cdb2662c6a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 37717 } } peers { permanent_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33337 } }
I20260926 14:52:30.258603 25831 ts_tablet_manager.cc:1428] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260926 14:52:30.258976 25660 heartbeater.cc:503] Master 127.12.68.254:42773 was elected leader, sending a full tablet report...
I20260926 14:52:30.260382 25833 tablet_bootstrap.cc:492] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872: Bootstrap starting.
I20260926 14:52:30.262661 25833 tablet_bootstrap.cc:654] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:30.265051 25833 log.cc:824] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:30.266090 25829 tablet_bootstrap.cc:492] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a: No bootstrap required, opened a new log
I20260926 14:52:30.266287 25829 ts_tablet_manager.cc:1397] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20260926 14:52:30.267316 25833 tablet_bootstrap.cc:492] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872: No bootstrap required, opened a new log
I20260926 14:52:30.267493 25833 ts_tablet_manager.cc:1397] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 14:52:30.270203 25829 raft_consensus.cc:348] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73eff62ef88244d293e8d919ef7bc4e8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45355 } } peers { permanent_uuid: "d8df6dc6da974461bb6810cdb2662c6a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 37717 } } peers { permanent_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33337 } }
I20260926 14:52:30.270452 25829 raft_consensus.cc:374] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:30.270504 25829 raft_consensus.cc:729] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8df6dc6da974461bb6810cdb2662c6a, State: Initialized, Role: FOLLOWER
I20260926 14:52:30.271312 25829 consensus_queue.cc:260] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73eff62ef88244d293e8d919ef7bc4e8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45355 } } peers { permanent_uuid: "d8df6dc6da974461bb6810cdb2662c6a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 37717 } } peers { permanent_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33337 } }
I20260926 14:52:30.271948 25833 raft_consensus.cc:348] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73eff62ef88244d293e8d919ef7bc4e8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45355 } } peers { permanent_uuid: "d8df6dc6da974461bb6810cdb2662c6a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 37717 } } peers { permanent_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33337 } }
I20260926 14:52:30.272243 25833 raft_consensus.cc:374] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:30.272327 25833 raft_consensus.cc:729] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9e753e43bc43dc8fb2b3ef5a6f4872, State: Initialized, Role: FOLLOWER
I20260926 14:52:30.272373 25829 ts_tablet_manager.cc:1428] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 14:52:30.272742 25794 heartbeater.cc:503] Master 127.12.68.254:42773 was elected leader, sending a full tablet report...
I20260926 14:52:30.272828 25833 consensus_queue.cc:260] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73eff62ef88244d293e8d919ef7bc4e8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45355 } } peers { permanent_uuid: "d8df6dc6da974461bb6810cdb2662c6a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 37717 } } peers { permanent_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33337 } }
I20260926 14:52:30.273756 25833 ts_tablet_manager.cc:1428] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 14:52:30.274103 25526 heartbeater.cc:503] Master 127.12.68.254:42773 was elected leader, sending a full tablet report...
W20260926 14:52:30.397379 25661 tablet.cc:2369] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:52:30.400138 25795 tablet.cc:2369] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:52:30.406574 25527 tablet.cc:2369] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:52:30.479394 25837 raft_consensus.cc:482] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:52:30.479610 25837 raft_consensus.cc:504] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73eff62ef88244d293e8d919ef7bc4e8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45355 } } peers { permanent_uuid: "d8df6dc6da974461bb6810cdb2662c6a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 37717 } } peers { permanent_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33337 } }
I20260926 14:52:30.479853 25836 raft_consensus.cc:482] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:52:30.480046 25836 raft_consensus.cc:504] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73eff62ef88244d293e8d919ef7bc4e8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45355 } } peers { permanent_uuid: "d8df6dc6da974461bb6810cdb2662c6a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 37717 } } peers { permanent_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33337 } }
I20260926 14:52:30.480808 25837 leader_election.cc:288] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73eff62ef88244d293e8d919ef7bc4e8 (127.12.68.194:45355), d8df6dc6da974461bb6810cdb2662c6a (127.12.68.195:37717)
I20260926 14:52:30.481030 25836 leader_election.cc:288] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73eff62ef88244d293e8d919ef7bc4e8 (127.12.68.194:45355), 2d9e753e43bc43dc8fb2b3ef5a6f4872 (127.12.68.193:33337)
I20260926 14:52:30.485818 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.481688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:40881 (local address 127.12.68.193:33337)
0926 14:52:30.481783 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:30.481810 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:30.482014 (+ 204us) server_negotiation.cc:407] Connection header received
0926 14:52:30.482191 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:30.482194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:30.482237 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:30.482281 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:30.482669 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.483214 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.484721 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.485284 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.485376 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.485423 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:30.485612 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 14:52:30.485687 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":35}
I20260926 14:52:30.486068 25826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.481155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:45055 (local address 127.12.68.194:45355)
0926 14:52:30.481284 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:30.481288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:30.481606 (+ 318us) server_negotiation.cc:407] Connection header received
0926 14:52:30.481818 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:30.481821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:30.481872 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:30.481933 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:30.482366 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.483178 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.484759 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.485448 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.485550 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.485596 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:30.485870 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 14:52:30.485955 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":52}
I20260926 14:52:30.486268 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.481181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:45355 (local address 127.12.68.195:37125)
0926 14:52:30.481667 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:30.481678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:30.481759 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:30.482276 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:30.482278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:30.482298 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:30.482435 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:30.482442 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.483344 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.483347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:30.484696 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:30.484705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.485858 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.485862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:30.485968 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.485989 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:52:30.486048 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 14:52:30.486108 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":4,"thread_start_us":95,"threads_started":1}
I20260926 14:52:30.486442 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.481065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:45355 (local address 127.12.68.193:45055)
0926 14:52:30.481515 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:30.481529 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:30.481693 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:30.482079 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:30.482081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:30.482100 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:30.482245 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:30.482255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.483311 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.483314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:30.484612 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:30.484622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.485645 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.485648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:30.485739 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.485760 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:52:30.486221 (+ 461us) client_negotiation.cc:240] Negotiation successful
0926 14:52:30.486291 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":115,"threads_started":1}
I20260926 14:52:30.486596 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.481649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:37125 (local address 127.12.68.194:45355)
0926 14:52:30.482002 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:30.482010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:30.482021 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:52:30.482057 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:30.482060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:30.482091 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:30.482158 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:30.482544 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.483243 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.484801 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.485667 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.485761 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.485808 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:30.486408 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 14:52:30.486475 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":193,"server-negotiator.queue_time_us":292,"thread_start_us":95,"threads_started":1}
I20260926 14:52:30.486788 25828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.481880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:33523 (local address 127.12.68.195:37717)
0926 14:52:30.481962 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:30.481966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:30.482181 (+ 215us) server_negotiation.cc:407] Connection header received
0926 14:52:30.482342 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:30.482345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:30.482448 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:30.482490 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:30.483478 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.484240 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.485682 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.486193 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.486264 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.486304 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:30.486626 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 14:52:30.486691 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":34,"spinlock_wait_cycles":31872}
I20260926 14:52:30.487025 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.481305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:33337 (local address 127.12.68.195:40881)
0926 14:52:30.481936 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:30.481948 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:30.482079 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:30.482396 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:30.482399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:30.482411 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:30.482563 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:30.482570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.483319 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.483322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:30.484602 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:30.484611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.485376 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.485379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:30.485479 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.485504 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:52:30.486229 (+ 725us) client_negotiation.cc:240] Negotiation successful
0926 14:52:30.486296 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":46,"thread_start_us":96,"threads_started":1}
I20260926 14:52:30.487058 25598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcae952fa404a60941f95c806c3f73a" candidate_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73eff62ef88244d293e8d919ef7bc4e8" is_pre_election: true
I20260926 14:52:30.487143 25848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.481453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:37717 (local address 127.12.68.193:33523)
0926 14:52:30.482110 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:30.482122 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:30.482245 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:30.482612 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:30.482614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:30.482629 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:30.483364 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:30.483371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.484444 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.484447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:30.485565 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:30.485573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.486392 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.486395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:30.486499 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.486519 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:52:30.486950 (+ 431us) client_negotiation.cc:240] Negotiation successful
0926 14:52:30.487015 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":52,"spinlock_wait_cycles":28032,"thread_start_us":84,"threads_started":1}
I20260926 14:52:30.487085 25614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcae952fa404a60941f95c806c3f73a" candidate_uuid: "d8df6dc6da974461bb6810cdb2662c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73eff62ef88244d293e8d919ef7bc4e8" is_pre_election: true
I20260926 14:52:30.487363 25598 raft_consensus.cc:2454] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d9e753e43bc43dc8fb2b3ef5a6f4872 in term 0.
I20260926 14:52:30.487638 25730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcae952fa404a60941f95c806c3f73a" candidate_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8df6dc6da974461bb6810cdb2662c6a" is_pre_election: true
I20260926 14:52:30.487818 25414 leader_election.cc:302] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d9e753e43bc43dc8fb2b3ef5a6f4872, 73eff62ef88244d293e8d919ef7bc4e8; no voters:
I20260926 14:52:30.487843 25730 raft_consensus.cc:2454] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d9e753e43bc43dc8fb2b3ef5a6f4872 in term 0.
I20260926 14:52:30.487972 25468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcae952fa404a60941f95c806c3f73a" candidate_uuid: "d8df6dc6da974461bb6810cdb2662c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" is_pre_election: true
I20260926 14:52:30.488158 25468 raft_consensus.cc:2454] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8df6dc6da974461bb6810cdb2662c6a in term 0.
I20260926 14:52:30.488272 25837 raft_consensus.cc:2781] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:52:30.488354 25837 raft_consensus.cc:482] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:52:30.488407 25837 raft_consensus.cc:3037] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:30.488545 25681 leader_election.cc:302] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d9e753e43bc43dc8fb2b3ef5a6f4872, d8df6dc6da974461bb6810cdb2662c6a; no voters: 73eff62ef88244d293e8d919ef7bc4e8
I20260926 14:52:30.488785 25836 raft_consensus.cc:2781] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:52:30.488857 25836 raft_consensus.cc:482] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:52:30.488903 25836 raft_consensus.cc:3037] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:30.489512 25837 raft_consensus.cc:504] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73eff62ef88244d293e8d919ef7bc4e8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45355 } } peers { permanent_uuid: "d8df6dc6da974461bb6810cdb2662c6a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 37717 } } peers { permanent_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33337 } }
I20260926 14:52:30.489828 25837 leader_election.cc:288] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [CANDIDATE]: Term 1 election: Requested vote from peers 73eff62ef88244d293e8d919ef7bc4e8 (127.12.68.194:45355), d8df6dc6da974461bb6810cdb2662c6a (127.12.68.195:37717)
I20260926 14:52:30.489835 25836 raft_consensus.cc:504] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73eff62ef88244d293e8d919ef7bc4e8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45355 } } peers { permanent_uuid: "d8df6dc6da974461bb6810cdb2662c6a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 37717 } } peers { permanent_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33337 } }
I20260926 14:52:30.490108 25614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcae952fa404a60941f95c806c3f73a" candidate_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73eff62ef88244d293e8d919ef7bc4e8"
I20260926 14:52:30.490144 25836 leader_election.cc:288] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [CANDIDATE]: Term 1 election: Requested vote from peers 73eff62ef88244d293e8d919ef7bc4e8 (127.12.68.194:45355), 2d9e753e43bc43dc8fb2b3ef5a6f4872 (127.12.68.193:33337)
I20260926 14:52:30.490149 25730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcae952fa404a60941f95c806c3f73a" candidate_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8df6dc6da974461bb6810cdb2662c6a"
I20260926 14:52:30.490216 25614 raft_consensus.cc:3037] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:30.490355 25730 raft_consensus.cc:2379] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2d9e753e43bc43dc8fb2b3ef5a6f4872 in current term 1: Already voted for candidate d8df6dc6da974461bb6810cdb2662c6a in this term.
I20260926 14:52:30.490504 25598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcae952fa404a60941f95c806c3f73a" candidate_uuid: "d8df6dc6da974461bb6810cdb2662c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73eff62ef88244d293e8d919ef7bc4e8"
I20260926 14:52:30.490597 25468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcae952fa404a60941f95c806c3f73a" candidate_uuid: "d8df6dc6da974461bb6810cdb2662c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872"
I20260926 14:52:30.490734 25468 raft_consensus.cc:2379] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d8df6dc6da974461bb6810cdb2662c6a in current term 1: Already voted for candidate 2d9e753e43bc43dc8fb2b3ef5a6f4872 in this term.
I20260926 14:52:30.491048 25681 leader_election.cc:302] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8df6dc6da974461bb6810cdb2662c6a; no voters: 2d9e753e43bc43dc8fb2b3ef5a6f4872, 73eff62ef88244d293e8d919ef7bc4e8
I20260926 14:52:30.491199 25836 raft_consensus.cc:2726] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 14:52:30.491382 25614 raft_consensus.cc:2454] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d9e753e43bc43dc8fb2b3ef5a6f4872 in term 1.
I20260926 14:52:30.491617 25414 leader_election.cc:302] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d9e753e43bc43dc8fb2b3ef5a6f4872, 73eff62ef88244d293e8d919ef7bc4e8; no voters: d8df6dc6da974461bb6810cdb2662c6a
I20260926 14:52:30.491760 25837 raft_consensus.cc:2781] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:52:30.492004 25837 raft_consensus.cc:686] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [term 1 LEADER]: Becoming Leader. State: Replica: 2d9e753e43bc43dc8fb2b3ef5a6f4872, State: Running, Role: LEADER
I20260926 14:52:30.492278 25837 consensus_queue.cc:237] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73eff62ef88244d293e8d919ef7bc4e8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45355 } } peers { permanent_uuid: "d8df6dc6da974461bb6810cdb2662c6a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 37717 } } peers { permanent_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33337 } }
I20260926 14:52:30.495194 25328 catalog_manager.cc:5495] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d9e753e43bc43dc8fb2b3ef5a6f4872 (127.12.68.193). New cstate: current_term: 1 leader_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73eff62ef88244d293e8d919ef7bc4e8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8df6dc6da974461bb6810cdb2662c6a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 37717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d9e753e43bc43dc8fb2b3ef5a6f4872" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 33337 } health_report { overall_health: HEALTHY } } }
I20260926 14:52:30.505841 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.502578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.12.68.193:33337)
0926 14:52:30.502687 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:30.502691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:30.502707 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:52:30.502863 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:30.502866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:30.502920 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:30.502999 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:52:30.503220 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.503721 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.504577 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:30.504784 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:30.504864 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:30.505133 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:52:30.505499 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:52:30.505597 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:30.505665 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 14:52:30.505710 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:52:30.510816 25614 raft_consensus.cc:1261] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8 [term 1 FOLLOWER]: Refusing update from remote peer 2d9e753e43bc43dc8fb2b3ef5a6f4872: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:52:30.510916 25730 raft_consensus.cc:1261] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 1 FOLLOWER]: Refusing update from remote peer 2d9e753e43bc43dc8fb2b3ef5a6f4872: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:52:30.511487 25850 consensus_queue.cc:1035] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73eff62ef88244d293e8d919ef7bc4e8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:52:30.511648 25837 consensus_queue.cc:1035] T 1bcae952fa404a60941f95c806c3f73a P 2d9e753e43bc43dc8fb2b3ef5a6f4872 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8df6dc6da974461bb6810cdb2662c6a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 37717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:52:30.518855 25854 mvcc.cc:205] Tried to move back new op lower bound from 7333619099686514688 to 7333619099618492416. Current Snapshot: MvccSnapshot[applied={T|T < 7333619099686514688}]
I20260926 14:52:30.519695 25855 mvcc.cc:205] Tried to move back new op lower bound from 7333619099686514688 to 7333619099618492416. Current Snapshot: MvccSnapshot[applied={T|T < 7333619099686514688}]
I20260926 14:52:30.526226 25328 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:52:30.526362 25328 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:52:30.528400 25328 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dccc4d93b2194db0a54c93e4346c5844: Sending DeleteTablet for 3 replicas of tablet 1bcae952fa404a60941f95c806c3f73a
I20260926 14:52:30.529095 25719 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bcae952fa404a60941f95c806c3f73a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:52:30 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:45496
I20260926 14:52:30.529150 25578 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bcae952fa404a60941f95c806c3f73a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:52:30 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:36528
I20260926 14:52:30.529505 25863 tablet_replica.cc:330] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a: stopping tablet replica
I20260926 14:52:30.529603 25459 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bcae952fa404a60941f95c806c3f73a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:52:30 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:44366
I20260926 14:52:30.529789 25863 raft_consensus.cc:2229] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:52:30.530071 25863 raft_consensus.cc:2258] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:52:30.530120 25864 tablet_replica.cc:330] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8: stopping tablet replica
I20260926 14:52:30.530143 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 25366
I20260926 14:52:30.530848 25864 raft_consensus.cc:2229] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:52:30.531154 25864 raft_consensus.cc:2258] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:52:30.532408 25864 ts_tablet_manager.cc:1905] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:52:30.532408 25863 ts_tablet_manager.cc:1905] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:52:30.534811 25863 ts_tablet_manager.cc:1918] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:52:30.534864 25864 ts_tablet_manager.cc:1918] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:52:30.534894 25863 log.cc:1196] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/1bcae952fa404a60941f95c806c3f73a
I20260926 14:52:30.534936 25864 log.cc:1196] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/1bcae952fa404a60941f95c806c3f73a
I20260926 14:52:30.535187 25863 ts_tablet_manager.cc:1939] T 1bcae952fa404a60941f95c806c3f73a P d8df6dc6da974461bb6810cdb2662c6a: Deleting consensus metadata
I20260926 14:52:30.535271 25864 ts_tablet_manager.cc:1939] T 1bcae952fa404a60941f95c806c3f73a P 73eff62ef88244d293e8d919ef7bc4e8: Deleting consensus metadata
I20260926 14:52:30.536145 25313 catalog_manager.cc:4841] TS d8df6dc6da974461bb6810cdb2662c6a (127.12.68.195:37717): tablet 1bcae952fa404a60941f95c806c3f73a (table test-table [id=fe3ef739dc4341c2ad46b863c7fc52c6]) successfully deleted
I20260926 14:52:30.536326 25313 catalog_manager.cc:4841] TS 73eff62ef88244d293e8d919ef7bc4e8 (127.12.68.194:45355): tablet 1bcae952fa404a60941f95c806c3f73a (table test-table [id=fe3ef739dc4341c2ad46b863c7fc52c6]) successfully deleted
W20260926 14:52:30.538569 25312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:33337 (error 108)
I20260926 14:52:30.539179 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 25530
I20260926 14:52:30.539455 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:30.539193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:33337 (local address 127.0.0.1:44388)
0926 14:52:30.539318 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:30.539384 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:33337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 14:52:30.539834 25312 catalog_manager.cc:4584] TS 2d9e753e43bc43dc8fb2b3ef5a6f4872 (127.12.68.193:33337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1bcae952fa404a60941f95c806c3f73a: Network error: Client connection negotiation failed: client connection to 127.12.68.193:33337: connect: Connection refused (error 111)
I20260926 14:52:30.543721 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 25663
I20260926 14:52:30.548069 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 25295
I20260926 14:52:30.553443 12563 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 14:52:30.887023 12563 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 14:52:30.887122 12563 mini_ranger.cc:66] Stopping Ranger...
I20260926 14:52:31.220870 12563 mini_ranger.cc:68] Stopped Ranger
2026-09-26 14:52:31.221 UTC [24362] LOG: received smart shutdown request
2026-09-26 14:52:31.224 UTC [24362] LOG: background worker "logical replication launcher" (PID 24371) exited with exit code 1
2026-09-26 14:52:31.224 UTC [24366] LOG: shutting down
2026-09-26 14:52:31.358 UTC [24362] LOG: database system is shut down
2026-09-26T14:52:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45986 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:52:31 dist-test-slave-24mn krb5kdc[25875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:52:31 dist-test-slave-24mn krb5kdc[25875](info): set up 2 sockets
Sep 26 14:52:31 dist-test-slave-24mn krb5kdc[25875](info): commencing operation
krb5kdc: starting...
W20260926 14:52:32.403587 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:52:32 dist-test-slave-24mn krb5kdc[25875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:52:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:52:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:52:32.508912 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:43649
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:43649,127.12.68.253:33395,127.12.68.252:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:32.612036 25888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:32.612283 25888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:32.612336 25888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:32.616264 25888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:52:32.616314 25888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:32.616339 25888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:32.616362 25888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:52:32.616390 25888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:52:32.621410 25888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:43649,127.12.68.253:33395,127.12.68.252:39107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:43649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:32.622362 25888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:32.623302 25888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:32.631320 25896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:32.631326 25893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:32.631325 25894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:32.631361 25888 server_base.cc:1034] running on GCE node
I20260926 14:52:32.632104 25888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:32.633147 25888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:32.634303 25888 hybrid_clock.cc:648] HybridClock initialized: now 1790434352634289 us; error 27 us; skew 500 ppm
Sep 26 14:52:32 dist-test-slave-24mn krb5kdc[25875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:32.636854 25888 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:52:32.637665 25888 webserver.cc:460] Webserver started at http://127.12.68.254:33033/ using document root <none> and password file <none>
I20260926 14:52:32.638145 25888 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:32.638199 25888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:32.638358 25888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:52:32.640262 25888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "db744b44910d436eb8d68d2987ba26e5"
format_stamp: "Formatted at 2026-09-26 14:52:32 on dist-test-slave-24mn"
server_key: "71f6d64536009452abcd521b91e705d5"
server_key_iv: "65636c9befcb7c9bcddf537b789badfa"
server_key_version: "encryptionkey@0"
I20260926 14:52:32.640758 25888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "db744b44910d436eb8d68d2987ba26e5"
format_stamp: "Formatted at 2026-09-26 14:52:32 on dist-test-slave-24mn"
server_key: "71f6d64536009452abcd521b91e705d5"
server_key_iv: "65636c9befcb7c9bcddf537b789badfa"
server_key_version: "encryptionkey@0"
I20260926 14:52:32.644124 25888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:52:32.646309 25903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:32.647483 25888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:52:32.647583 25888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "db744b44910d436eb8d68d2987ba26e5"
format_stamp: "Formatted at 2026-09-26 14:52:32 on dist-test-slave-24mn"
server_key: "71f6d64536009452abcd521b91e705d5"
server_key_iv: "65636c9befcb7c9bcddf537b789badfa"
server_key_version: "encryptionkey@0"
I20260926 14:52:32.647671 25888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:32.666936 25888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:32.669667 25888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:32.669840 25888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:32.676398 25888 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:43649
I20260926 14:52:32.676416 25955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:43649 every 8 connection(s)
I20260926 14:52:32.677284 25888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:52:32.680027 25956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:32.683838 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 25888
I20260926 14:52:32.683979 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:52:32.683609 25956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:52:32.684286 12563 external_mini_cluster.cc:1428] Setting key 5bdcfc6f1c2abe7881e77831bbcd2fff
WARNING: no policy specified for kudu/127.12.68.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.253@KRBTEST.COM" created.
Sep 26 14:52:32 dist-test-slave-24mn krb5kdc[25875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:32.699499 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.686147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.12.68.254:43649)
0926 14:52:32.686693 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:32.686711 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:32.686853 (+ 142us) server_negotiation.cc:407] Connection header received
0926 14:52:32.688023 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:32.688028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:32.688200 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:32.688342 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:32.689359 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.690237 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.691628 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.692044 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.692153 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:32.695252 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:32.695257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:32.695266 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:32.695286 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:32.696716 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:32.697227 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:32.697231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:32.697234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:32.697327 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:32.697664 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:32.697666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:32.697668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:32.698002 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:32.698095 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:32.698510 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 14:52:32.698733 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":425,"spinlock_wait_cycles":49408,"thread_start_us":148,"threads_started":1}
I20260926 14:52:32.699502 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.686147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43649 (local address 127.0.0.1:48870)
0926 14:52:32.686686 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.686737 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:32.687705 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:32.688396 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:32.688402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:32.688833 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:32.689175 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:32.689195 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.690358 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.690362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:32.691522 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:32.691534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.692192 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.692197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:32.692391 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:32.692909 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:32.692924 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:32.695125 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:32.696831 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:32.696840 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:32.696849 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:32.697129 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:32.697416 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:32.697420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:32.697421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:32.697563 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:32.698125 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:32.698131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:32.698286 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 14:52:32.698469 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 14:52:32.698804 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":97,"thread_start_us":148,"threads_started":1}
I20260926 14:52:32.701843 25956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:52:32.702498 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.702244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:33395 (local address 127.0.0.1:39588)
0926 14:52:32.702364 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.702434 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:33395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 14:52:32.702670 25907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.68.253:33395: connect: Connection refused (error 111)
I20260926 14:52:32.703420 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.703231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:33395 (local address 127.0.0.1:39600)
0926 14:52:32.703330 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.703374 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:33395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 14:52:32.703540 25956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.253:33395: Network error: Client connection negotiation failed: client connection to 127.12.68.253:33395: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.253.keytab.
Entry for principal kudu/127.12.68.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.253.keytab.
I20260926 14:52:32.708304 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.253:33395
--webserver_interface=127.12.68.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.253.keytab
--principal=kudu/127.12.68.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:43649,127.12.68.253:33395,127.12.68.252:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:52:32.752770 25956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } attempt: 1
I20260926 14:52:32.753625 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.753297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:33395 (local address 127.0.0.1:39608)
0926 14:52:32.753461 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.753532 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:33395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 14:52:32.754317 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.754084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:33395 (local address 127.0.0.1:39620)
0926 14:52:32.754183 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.754238 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:33395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 14:52:32.754453 25956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.253:33395: Network error: Client connection negotiation failed: client connection to 127.12.68.253:33395: connect: Connection refused (error 111)
W20260926 14:52:32.814558 25963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:32.814800 25963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:32.814839 25963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:32.818715 25963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:52:32.818766 25963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:32.818790 25963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:32.818805 25963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:52:32.818818 25963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:52:32.822610 25956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } attempt: 2
I20260926 14:52:32.823370 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.823072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:33395 (local address 127.0.0.1:39634)
0926 14:52:32.823216 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.823286 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:33395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 14:52:32.823988 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.823769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:33395 (local address 127.0.0.1:39640)
0926 14:52:32.823856 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.823924 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:33395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 14:52:32.824105 25956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.253:33395: Network error: Client connection negotiation failed: client connection to 127.12.68.253:33395: connect: Connection refused (error 111)
I20260926 14:52:32.824136 25963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:43649,127.12.68.253:33395,127.12.68.252:39107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.253.keytab
--principal=kudu/127.12.68.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.253:33395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.68.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:32.825032 25963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:32.825927 25963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:32.833673 25969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:32.833643 25968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:32.833670 25971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:32.834054 25963 server_base.cc:1034] running on GCE node
I20260926 14:52:32.834568 25963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:32.835595 25963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:32.836750 25963 hybrid_clock.cc:648] HybridClock initialized: now 1790434352836735 us; error 28 us; skew 500 ppm
Sep 26 14:52:32 dist-test-slave-24mn krb5kdc[25875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:32.839334 25963 init.cc:377] Logged in from keytab as kudu/127.12.68.253@KRBTEST.COM (short username kudu)
I20260926 14:52:32.840196 25963 webserver.cc:460] Webserver started at http://127.12.68.253:40429/ using document root <none> and password file <none>
I20260926 14:52:32.840695 25963 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:32.840751 25963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:32.840950 25963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:52:32.842893 25963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/data/instance:
uuid: "3a18302afea342168d4c86d1cd49b5b1"
format_stamp: "Formatted at 2026-09-26 14:52:32 on dist-test-slave-24mn"
server_key: "a853c639a915fd2f40feb94071e7b4c1"
server_key_iv: "2c4e3197103148f36322589505d44d57"
server_key_version: "encryptionkey@0"
I20260926 14:52:32.843416 25963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/wal/instance:
uuid: "3a18302afea342168d4c86d1cd49b5b1"
format_stamp: "Formatted at 2026-09-26 14:52:32 on dist-test-slave-24mn"
server_key: "a853c639a915fd2f40feb94071e7b4c1"
server_key_iv: "2c4e3197103148f36322589505d44d57"
server_key_version: "encryptionkey@0"
I20260926 14:52:32.847011 25963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:52:32.849818 25978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:32.851030 25963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:52:32.851145 25963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/wal
uuid: "3a18302afea342168d4c86d1cd49b5b1"
format_stamp: "Formatted at 2026-09-26 14:52:32 on dist-test-slave-24mn"
server_key: "a853c639a915fd2f40feb94071e7b4c1"
server_key_iv: "2c4e3197103148f36322589505d44d57"
server_key_version: "encryptionkey@0"
I20260926 14:52:32.851249 25963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:32.871341 25963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:32.874051 25963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:32.874238 25963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:32.880673 25963 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.253:33395
I20260926 14:52:32.880725 26030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.253:33395 every 8 connection(s)
I20260926 14:52:32.881564 25963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/data/info.pb
I20260926 14:52:32.883320 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 25963
I20260926 14:52:32.883447 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-1/wal/instance
I20260926 14:52:32.883744 12563 external_mini_cluster.cc:1428] Setting key 8279ec13833fd7056ad4936a5bcd9eeb
I20260926 14:52:32.884555 26031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:32.888347 26031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.68.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.252@KRBTEST.COM" created.
Sep 26 14:52:32 dist-test-slave-24mn krb5kdc[25875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.253@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:32.901934 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.890161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.12.68.254:43649)
0926 14:52:32.890278 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:32.890281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:32.891289 (+ 1008us) server_negotiation.cc:407] Connection header received
0926 14:52:32.892187 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:32.892191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:32.892254 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:32.892323 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:32.893423 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.893953 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.895474 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.895814 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.895942 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:32.898534 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:32.898538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:32.898541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:32.898570 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:32.900122 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:32.900570 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:32.900574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:32.900576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:32.900665 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:32.900929 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:32.900931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:32.900932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:32.901104 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:32.901175 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:32.901694 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 14:52:32.901814 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:52:32.902808 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.890670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43649 (local address 127.0.0.1:48882)
0926 14:52:32.891149 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.891189 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:32.891994 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:32.892493 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:32.892500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:32.892803 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:32.893289 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:32.893303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.894069 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.894073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:32.895349 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:32.895361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.895998 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.896001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:32.896152 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:32.896718 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:32.896733 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:32.898361 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:32.900224 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:32.900229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:32.900240 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:32.900479 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:32.900739 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:32.900741 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:32.900742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:32.900863 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:32.901188 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:32.901194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:32.901490 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 14:52:32.901640 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 14:52:32.901934 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":137,"threads_started":1}
I20260926 14:52:32.904119 26031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.252.keytab.
Entry for principal kudu/127.12.68.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.252.keytab.
I20260926 14:52:32.908064 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.252:39107
--webserver_interface=127.12.68.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.252.keytab
--principal=kudu/127.12.68.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:43649,127.12.68.253:33395,127.12.68.252:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:52:32 dist-test-slave-24mn krb5kdc[25875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.253@KRBTEST.COM for kudu/127.12.68.253@KRBTEST.COM
I20260926 14:52:32.912389 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.904553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:33395 (local address 127.0.0.1:39654)
0926 14:52:32.904659 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.904672 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:32.904728 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:32.905301 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:32.905304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:32.905464 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:32.905540 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:32.905545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.906506 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.906509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:32.907088 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:32.907097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.907706 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.907708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:32.907823 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:32.908280 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:32.908291 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:32.909417 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:32.911076 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:32.911078 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:32.911080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:32.911237 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:32.911480 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:32.911483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:32.911485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:32.911554 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:32.911942 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:32.911945 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:32.912026 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 14:52:32.912111 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 14:52:32.912226 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":25}
I20260926 14:52:32.912436 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.904648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.12.68.253:33395)
0926 14:52:32.904960 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:32.904969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:32.904990 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:52:32.905020 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:32.905024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:32.905161 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:32.905325 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:32.905632 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.906412 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.907175 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.907599 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.907699 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:32.909527 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:32.909536 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:32.909549 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:32.909569 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:32.910917 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:32.911331 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:32.911335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:32.911336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:32.911404 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:32.911623 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:32.911626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:32.911628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:32.911829 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:32.911964 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:32.912196 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 14:52:32.912310 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":235,"thread_start_us":102,"threads_started":1}
I20260926 14:52:32.914063 26031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:52:32.914641 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.914425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51108)
0926 14:52:32.914521 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.914590 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 14:52:32.914803 25980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
I20260926 14:52:32.915248 25956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } attempt: 3
I20260926 14:52:32.915591 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.915370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51118)
0926 14:52:32.915488 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.915531 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 14:52:32.915966 26031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:39107: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
Sep 26 14:52:32 dist-test-slave-24mn krb5kdc[25875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for kudu/127.12.68.253@KRBTEST.COM
I20260926 14:52:32.922757 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.915635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:33395 (local address 127.0.0.1:39664)
0926 14:52:32.915799 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.915812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:32.915919 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:32.916189 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:32.916191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:32.916322 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:32.916433 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:32.916438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.917395 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.917397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:32.917899 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:32.917905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.918453 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.918456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:32.918527 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:32.918931 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:32.918941 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:32.920053 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:32.921627 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:32.921630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:32.921631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:32.921799 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:32.922037 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:32.922039 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:32.922040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:32.922101 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:32.922414 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:32.922416 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:32.922488 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 14:52:32.922557 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:52:32.922648 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 14:52:32.922854 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.915693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.12.68.253:33395)
0926 14:52:32.915792 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:32.915797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:32.915869 (+ 72us) server_negotiation.cc:407] Connection header received
0926 14:52:32.915995 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:32.915997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:32.916043 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:32.916167 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:32.916535 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.917308 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.917993 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:32.918366 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:32.918459 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:32.920171 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:32.920175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:32.920178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:32.920202 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:32.921536 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:32.921876 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:32.921880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:32.921881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:32.921964 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:32.922163 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:32.922165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:32.922167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:32.922344 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:32.922428 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:32.922616 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 14:52:32.922718 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:52:32.924026 25956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:52:32.924645 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.924418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51130)
0926 14:52:32.924529 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.924580 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 14:52:32.925393 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.925164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51142)
0926 14:52:32.925288 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.925336 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260926 14:52:32.925531 25956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:39107: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
I20260926 14:52:32.956670 25956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } attempt: 1
I20260926 14:52:32.957446 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.957127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51156)
0926 14:52:32.957285 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.957353 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 14:52:32.958129 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.957903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51168)
0926 14:52:32.958015 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.958066 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 14:52:32.958251 25956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:39107: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
I20260926 14:52:32.965193 26031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } attempt: 1
I20260926 14:52:32.965816 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.965589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51182)
0926 14:52:32.965702 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.965753 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 14:52:32.966481 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:32.966293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51190)
0926 14:52:32.966386 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:32.966429 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 14:52:32.966606 26031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:39107: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
W20260926 14:52:33.016068 26037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:33.016307 26037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:33.016345 26037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:33.020224 26037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:52:33.020269 26037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:33.020287 26037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:33.020301 26037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:52:33.020314 26037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:52:33.025326 26037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:43649,127.12.68.253:33395,127.12.68.252:39107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.252.keytab
--principal=kudu/127.12.68.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.252:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.68.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:33.026252 26037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:33.027172 26037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:52:33.033388 25956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } attempt: 2
I20260926 14:52:33.034220 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.033834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51204)
0926 14:52:33.034064 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:33.034124 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260926 14:52:33.034757 26031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } attempt: 2
I20260926 14:52:33.035094 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.034821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51206)
0926 14:52:33.034972 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:33.035027 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 14:52:33.035194 26043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:33.035400 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.035108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51216)
0926 14:52:33.035280 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:33.035331 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260926 14:52:33.035385 25956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:39107: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
I20260926 14:52:33.035392 26037 server_base.cc:1034] running on GCE node
W20260926 14:52:33.035619 26046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:33.035620 26044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:33.036060 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.035792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51218)
0926 14:52:33.035945 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:33.035998 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 14:52:33.036178 26031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:39107: Network error: Client connection negotiation failed: client connection to 127.12.68.252:39107: connect: Connection refused (error 111)
I20260926 14:52:33.036242 26037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:33.037268 26037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:33.038427 26037 hybrid_clock.cc:648] HybridClock initialized: now 1790434353038417 us; error 25 us; skew 500 ppm
Sep 26 14:52:33 dist-test-slave-24mn krb5kdc[25875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434353, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:33.040908 26037 init.cc:377] Logged in from keytab as kudu/127.12.68.252@KRBTEST.COM (short username kudu)
I20260926 14:52:33.041734 26037 webserver.cc:460] Webserver started at http://127.12.68.252:36111/ using document root <none> and password file <none>
I20260926 14:52:33.042230 26037 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:33.042292 26037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:33.042478 26037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:52:33.044448 26037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/data/instance:
uuid: "177d893e669d49e0bc1654adb0b1c961"
format_stamp: "Formatted at 2026-09-26 14:52:33 on dist-test-slave-24mn"
server_key: "b58872538dcb355f4ece4cfa409f1c2e"
server_key_iv: "aeeecc24f9bdff33eb125c496352d931"
server_key_version: "encryptionkey@0"
I20260926 14:52:33.044953 26037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/wal/instance:
uuid: "177d893e669d49e0bc1654adb0b1c961"
format_stamp: "Formatted at 2026-09-26 14:52:33 on dist-test-slave-24mn"
server_key: "b58872538dcb355f4ece4cfa409f1c2e"
server_key_iv: "aeeecc24f9bdff33eb125c496352d931"
server_key_version: "encryptionkey@0"
I20260926 14:52:33.048388 26037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:52:33.050601 26053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:33.051719 26037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:52:33.051824 26037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/wal
uuid: "177d893e669d49e0bc1654adb0b1c961"
format_stamp: "Formatted at 2026-09-26 14:52:33 on dist-test-slave-24mn"
server_key: "b58872538dcb355f4ece4cfa409f1c2e"
server_key_iv: "aeeecc24f9bdff33eb125c496352d931"
server_key_version: "encryptionkey@0"
I20260926 14:52:33.051945 26037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:33.075038 26037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:33.077787 26037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:33.077955 26037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:33.084574 26037 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.252:39107
I20260926 14:52:33.084586 26105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.252:39107 every 8 connection(s)
I20260926 14:52:33.085453 26037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/data/info.pb
I20260926 14:52:33.088018 26106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:33.091492 26106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:52:33.093638 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 26037
I20260926 14:52:33.093760 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434204424021-12563-0/minicluster-data/master-2/wal/instance
I20260926 14:52:33.094056 12563 external_mini_cluster.cc:1428] Setting key 9fa25879a7e11f7564e466d06ab53604
Sep 26 14:52:33 dist-test-slave-24mn krb5kdc[25875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434353, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.252@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
Sep 26 14:52:33 dist-test-slave-24mn krb5kdc[25875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.252@KRBTEST.COM
I20260926 14:52:33.103811 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.093232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.12.68.254:43649)
0926 14:52:33.093375 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:33.093379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:33.094203 (+ 824us) server_negotiation.cc:407] Connection header received
0926 14:52:33.095251 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:33.095255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:33.095310 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:33.095379 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:33.096414 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.096918 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.097833 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.098042 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.098127 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.100452 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:33.100455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:33.100457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:33.100482 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:33.101870 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.102429 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.102432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.102433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.102498 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.102805 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.102808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.102810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.102996 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:33.103080 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:33.103542 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 14:52:33.103655 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:52:33.104596 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.093617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:43649 (local address 127.0.0.1:48890)
0926 14:52:33.094069 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:33.094114 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:33.095052 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:33.095470 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:33.095479 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:33.095801 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:33.096269 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:33.096281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.097010 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.097014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:33.097720 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:33.097728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.098163 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.098167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:33.098339 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.098853 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:33.098871 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:33.100313 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:33.101969 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:33.101976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:33.101984 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:33.102349 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:33.102583 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:33.102586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:33.102588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:33.102719 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:33.103091 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:33.103098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:33.103321 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 14:52:33.103470 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 14:52:33.103753 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"spinlock_wait_cycles":1280,"thread_start_us":131,"threads_started":1}
I20260926 14:52:33.104601 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.094675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.12.68.252:39107)
0926 14:52:33.094974 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:33.094983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:33.095237 (+ 254us) server_negotiation.cc:407] Connection header received
0926 14:52:33.095433 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:33.095437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:33.095603 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:33.095753 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:33.096521 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.097329 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.098101 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.098447 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.098535 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.100821 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:33.100826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:33.100835 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:33.100854 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:33.102431 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.102779 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.102784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.102786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.102855 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.103085 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.103088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.103089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.103325 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:33.103494 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:33.103685 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 14:52:33.103811 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"spinlock_wait_cycles":14208,"thread_start_us":107,"threads_started":1}
I20260926 14:52:33.105741 26106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:52:33 dist-test-slave-24mn krb5kdc[25875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434353, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.252@KRBTEST.COM for kudu/127.12.68.253@KRBTEST.COM
Sep 26 14:52:33 dist-test-slave-24mn krb5kdc[25875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:33.113598 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.106140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:33395 (local address 127.0.0.1:39676)
0926 14:52:33.106454 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:33.106467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:33.106541 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:33.106781 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:33.106784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:33.106935 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:33.107047 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:33.107053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.107927 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.107930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:33.108533 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:33.108541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.109026 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.109029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:33.109124 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.109542 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:33.109552 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:33.110771 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:33.112383 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:33.112386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:33.112388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:33.112558 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:33.112796 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:33.112799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:33.112801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:33.112871 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:33.113184 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:33.113188 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:33.113268 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 14:52:33.113349 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 14:52:33.113458 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260926 14:52:33.113629 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.106244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.12.68.253:33395)
0926 14:52:33.106351 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:33.106356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:33.106519 (+ 163us) server_negotiation.cc:407] Connection header received
0926 14:52:33.106621 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:33.106624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:33.106675 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:33.106756 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:33.107162 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.107818 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.108632 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.108937 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.109018 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.110894 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:33.110897 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:33.110900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:33.110924 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:33.112285 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.112638 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.112642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.112643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.112714 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.112942 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.112945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.112947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.113105 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:33.113183 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:33.113405 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 14:52:33.113502 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 14:52:33.114108 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.106609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.12.68.254:43649)
0926 14:52:33.106720 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:33.106723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:33.106792 (+ 69us) server_negotiation.cc:407] Connection header received
0926 14:52:33.106894 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:33.106897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:33.106937 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:33.107013 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:33.107573 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.108124 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.108964 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.109252 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.109334 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.111454 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:33.111457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:33.111459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:33.111480 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:33.112725 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.113084 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.113087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.113088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.113155 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.113383 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.113386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.113387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.113556 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:33.113745 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:33.113868 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 14:52:33.113978 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:52:33.114509 26106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:52:33 dist-test-slave-24mn krb5kdc[25875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434353, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.252@KRBTEST.COM for kudu/127.12.68.252@KRBTEST.COM
Sep 26 14:52:33 dist-test-slave-24mn krb5kdc[25875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.253@KRBTEST.COM
I20260926 14:52:33.122011 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.114889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51244)
0926 14:52:33.114970 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:33.114981 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:33.115043 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:33.115246 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:33.115249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:33.115392 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:33.115481 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:33.115488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.116373 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.116376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:33.117037 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:33.117045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.117570 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.117573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:33.117650 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.118051 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:33.118063 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:33.119126 (+ 1063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:33.120725 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:33.120728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:33.120730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:33.120964 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:33.121190 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:33.121193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:33.121195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:33.121280 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:33.121568 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:33.121571 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:33.121644 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 14:52:33.121724 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 14:52:33.121865 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 14:52:33.122032 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.114947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.12.68.252:39107)
0926 14:52:33.115029 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:33.115032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:33.115043 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:52:33.115130 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:33.115133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:33.115164 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:33.115225 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:33.115574 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.116289 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.117131 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.117469 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.117553 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.119248 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:33.119251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:33.119254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:33.119277 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:33.120626 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.121047 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.121049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.121050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.121112 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.121353 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.121355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.121356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.121486 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:33.121564 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:33.121772 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 14:52:33.121917 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":35}
I20260926 14:52:33.122926 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.115243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.12.68.253:33395)
0926 14:52:33.115343 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:33.115347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:33.115359 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:52:33.115389 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:33.115391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:33.115426 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:33.115503 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:33.116132 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.116730 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.117558 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.117845 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.117919 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.120005 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:33.120008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:33.120011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:33.120033 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:33.121362 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.121699 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.121702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.121704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.121775 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.122006 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.122009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.122011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.122209 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:33.122395 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:33.122693 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 14:52:33.122791 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:52:33.125165 26106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961: Bootstrap starting.
I20260926 14:52:33.125859 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:52:33.127334 26031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } attempt: 3
I20260926 14:52:33.127451 26106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:33.128382 26106 log.cc:824] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:33.132510 26106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961: No bootstrap required, opened a new log
Sep 26 14:52:33 dist-test-slave-24mn krb5kdc[25875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.253@KRBTEST.COM for kudu/127.12.68.252@KRBTEST.COM
I20260926 14:52:33.134523 25956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } attempt: 3
I20260926 14:52:33.136565 26106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } }
I20260926 14:52:33.136813 26106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:33.137777 26106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 177d893e669d49e0bc1654adb0b1c961, State: Initialized, Role: FOLLOWER
I20260926 14:52:33.138463 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.128347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.12.68.253:33395)
0926 14:52:33.128435 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:33.128451 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:33.130046 (+ 1595us) server_negotiation.cc:407] Connection header received
0926 14:52:33.130600 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:33.130603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:33.130645 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:33.130694 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:33.131545 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.132254 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.133301 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.133588 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.133659 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.135088 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:33.135091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:33.135094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:33.135116 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:33.136628 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.137230 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.137234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.137236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.137308 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.137588 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.137591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.137593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.137777 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:33.137880 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:33.138219 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 14:52:33.138325 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:52:33.137077 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.128049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51268 (local address 127.12.68.252:39107)
0926 14:52:33.128848 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:33.128851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:33.128884 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:52:33.129011 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:33.129014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:33.129052 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:33.129126 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:33.129853 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.130483 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.131540 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.131843 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.131994 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.133433 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:33.133436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:33.133451 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:33.133474 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:33.134987 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.135384 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.135388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.135390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.135469 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.135753 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.135756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.135758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.136040 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:33.136156 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:33.136675 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 14:52:33.136912 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":78,"threads_started":1}
I20260926 14:52:33.138931 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.127971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51260 (local address 127.12.68.252:39107)
0926 14:52:33.128307 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:33.128315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:33.128326 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:52:33.128358 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:33.128361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:33.128403 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:33.128908 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:33.129008 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.129729 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.130727 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.131081 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.131165 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.133602 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:33.133606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:33.133608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:33.133630 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:33.137284 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.137703 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.137707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.137708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.137778 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.138046 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.138049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.138051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.138234 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:33.138310 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:33.138669 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 14:52:33.138779 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
I20260926 14:52:33.139756 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.127666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51260)
0926 14:52:33.127775 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:33.127799 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:33.127866 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:33.128533 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:33.128536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:33.128704 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:33.128799 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:33.128806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.129844 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.129847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:33.130602 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:33.130613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.131277 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.131281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:33.131358 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.131804 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:33.131818 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:33.133470 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:33.137381 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:33.137384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:33.137386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:33.137608 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:33.137870 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:33.137872 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:33.137874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:33.137952 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:33.138348 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:33.138351 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:33.138429 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 14:52:33.139084 (+ 655us) client_negotiation.cc:240] Negotiation successful
0926 14:52:33.139206 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 14:52:33.141743 26106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } }
I20260926 14:52:33.142390 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.128028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.12.68.254:43649)
0926 14:52:33.128131 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:33.128135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:33.128373 (+ 238us) server_negotiation.cc:407] Connection header received
0926 14:52:33.128580 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:33.128583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:33.128627 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:33.128675 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:33.129477 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.130028 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.136959 (+ 6931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.137286 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.137365 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.138781 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:33.138784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:33.138787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:33.138810 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:33.140524 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.140939 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.140943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.140945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.141024 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.141479 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.141483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.141484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.141714 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:33.141834 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:33.142154 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 14:52:33.142257 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 14:52:33 dist-test-slave-24mn krb5kdc[25875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for kudu/127.12.68.252@KRBTEST.COM
I20260926 14:52:33.143086 26031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1: Bootstrap starting.
I20260926 14:52:33.143203 26106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:52:33.145866 26031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:33.146016 26121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } } }
I20260926 14:52:33.146136 26121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:52:33.146740 26031 log.cc:824] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1: Log is configured to *not* fsync() on all Append() calls
W20260926 14:52:33.148594 26132 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:52:33.148711 26132 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:52:33.148936 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.134867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:39107 (local address 127.0.0.1:51280)
0926 14:52:33.135074 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:33.135086 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:33.135154 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:33.137704 (+ 2550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:33.137707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:33.138095 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:33.138193 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:33.138199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.139439 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.139442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:33.140447 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:33.140458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.141023 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.141027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:33.141110 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.141480 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:33.141493 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:33.142847 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:33.145633 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:33.145636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:33.145638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:33.145865 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:33.146179 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:33.146182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:33.146184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:33.146287 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:33.148452 (+ 2165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:33.148455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:33.148553 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 14:52:33.148654 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 14:52:33.148779 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"spinlock_wait_cycles":4352}
I20260926 14:52:33.146219 26037 master_runner.cc:186] Error getting master registration for 127.12.68.254:43649: OK, instance_id { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" instance_seqno: 1790434352674546 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:52:33.149255 26037 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:52:33.149449 26120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:33.135565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51280 (local address 127.12.68.252:39107)
0926 14:52:33.137490 (+ 1925us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:33.137499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:33.137509 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:52:33.137536 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:33.137538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:33.137575 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:33.138304 (+ 729us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:33.138388 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.139323 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.140587 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:33.140914 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:33.141001 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:33.143131 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:33.143135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:33.143137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:33.143161 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:33.145517 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.145987 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.145990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.145992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.146075 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:33.148038 (+ 1963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:33.148041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:33.148042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:33.148358 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:33.148451 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:33.149129 (+ 678us) server_negotiation.cc:299] Negotiation successful
0926 14:52:33.149235 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1863,"thread_start_us":81,"threads_started":1}
I20260926 14:52:33.150255 26031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1: No bootstrap required, opened a new log
I20260926 14:52:33.152256 25956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5: Bootstrap starting.
I20260926 14:52:33.154589 26031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } }
I20260926 14:52:33.154826 26031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:33.154899 26031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a18302afea342168d4c86d1cd49b5b1, State: Initialized, Role: FOLLOWER
I20260926 14:52:33.155041 25956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:33.156083 25956 log.cc:824] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:33.156267 26031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } }
I20260926 14:52:33.156729 26136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } } }
I20260926 14:52:33.158156 25956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5: No bootstrap required, opened a new log
I20260926 14:52:33.160004 26136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:52:33.157451 26031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:52:33.162011 25956 raft_consensus.cc:348] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } }
I20260926 14:52:33.162281 25956 raft_consensus.cc:374] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:33.162365 25956 raft_consensus.cc:729] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db744b44910d436eb8d68d2987ba26e5, State: Initialized, Role: FOLLOWER
I20260926 14:52:33.162770 26136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:52:33.162834 25956 consensus_queue.cc:260] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } }
I20260926 14:52:33.162907 26136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } }
I20260926 14:52:33.163337 26145 sys_catalog.cc:455] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } } }
I20260926 14:52:33.163586 26145 sys_catalog.cc:458] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:52:33.163820 26136 leader_election.cc:288] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db744b44910d436eb8d68d2987ba26e5 (127.12.68.254:43649), 177d893e669d49e0bc1654adb0b1c961 (127.12.68.252:39107)
I20260926 14:52:33.164059 25956 sys_catalog.cc:564] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:52:33.164686 26080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a18302afea342168d4c86d1cd49b5b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "177d893e669d49e0bc1654adb0b1c961" is_pre_election: true
I20260926 14:52:33.164959 26080 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a18302afea342168d4c86d1cd49b5b1 in term 0.
I20260926 14:52:33.165418 25930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a18302afea342168d4c86d1cd49b5b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db744b44910d436eb8d68d2987ba26e5" is_pre_election: true
I20260926 14:52:33.165647 25930 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a18302afea342168d4c86d1cd49b5b1 in term 0.
I20260926 14:52:33.166087 25980 leader_election.cc:302] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 177d893e669d49e0bc1654adb0b1c961, 3a18302afea342168d4c86d1cd49b5b1; no voters:
I20260926 14:52:33.166522 26136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:52:33.166625 26136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:52:33.166779 26136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:33.168195 26136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } }
I20260926 14:52:33.168601 26136 leader_election.cc:288] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [CANDIDATE]: Term 1 election: Requested vote from peers db744b44910d436eb8d68d2987ba26e5 (127.12.68.254:43649), 177d893e669d49e0bc1654adb0b1c961 (127.12.68.252:39107)
I20260926 14:52:33.169870 26080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a18302afea342168d4c86d1cd49b5b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "177d893e669d49e0bc1654adb0b1c961"
I20260926 14:52:33.169970 26080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:33.171005 26080 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a18302afea342168d4c86d1cd49b5b1 in term 1.
W20260926 14:52:33.171329 25909 tablet.cc:2369] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:52:33.172070 25930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a18302afea342168d4c86d1cd49b5b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db744b44910d436eb8d68d2987ba26e5"
I20260926 14:52:33.172165 25930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:33.173153 25888 master_runner.cc:263] No leader master found from master db744b44910d436eb8d68d2987ba26e5
I20260926 14:52:33.173209 25888 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:52:33.173615 25980 leader_election.cc:302] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 177d893e669d49e0bc1654adb0b1c961, 3a18302afea342168d4c86d1cd49b5b1; no voters:
I20260926 14:52:33.173717 25930 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a18302afea342168d4c86d1cd49b5b1 in term 1.
I20260926 14:52:33.175015 25963 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260926 14:52:33.175115 25963 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:52:33.175196 26136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:52:33.175348 26136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [term 1 LEADER]: Becoming Leader. State: Replica: 3a18302afea342168d4c86d1cd49b5b1, State: Running, Role: LEADER
I20260926 14:52:33.175616 26136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } }
W20260926 14:52:33.176819 26148 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:52:33.176877 26148 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:52:33.177160 26136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a18302afea342168d4c86d1cd49b5b1. Latest consensus state: current_term: 1 leader_uuid: "3a18302afea342168d4c86d1cd49b5b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } } }
I20260926 14:52:33.177242 26136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:33.177707 26164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:52:33.178509 26164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 14:52:33.178548 26159 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:52:33.178624 26159 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:52:33.183219 26080 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [term 1 FOLLOWER]: Refusing update from remote peer 3a18302afea342168d4c86d1cd49b5b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:52:33.183775 26161 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:52:33.186969 26121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a18302afea342168d4c86d1cd49b5b1. Latest consensus state: current_term: 1 leader_uuid: "3a18302afea342168d4c86d1cd49b5b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } } }
I20260926 14:52:33.187068 26121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:52:33.183223 25930 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [term 1 FOLLOWER]: Refusing update from remote peer 3a18302afea342168d4c86d1cd49b5b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:52:33.188603 26136 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:52:33.189396 26161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a18302afea342168d4c86d1cd49b5b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } } }
I20260926 14:52:33.189494 26161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:33.190641 26121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a18302afea342168d4c86d1cd49b5b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } } }
I20260926 14:52:33.190732 26121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:52:33.191139 26166 mvcc.cc:205] Tried to move back new op lower bound from 7333619110633148416 to 7333619110609920000. Current Snapshot: MvccSnapshot[applied={T|T < 7333619110633148416}]
I20260926 14:52:33.192273 26145 sys_catalog.cc:455] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a18302afea342168d4c86d1cd49b5b1. Latest consensus state: current_term: 1 leader_uuid: "3a18302afea342168d4c86d1cd49b5b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } } }
I20260926 14:52:33.192389 26145 sys_catalog.cc:458] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:52:33.192471 26145 sys_catalog.cc:455] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a18302afea342168d4c86d1cd49b5b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } } }
I20260926 14:52:33.192538 26145 sys_catalog.cc:458] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:52:33.194340 26164 catalog_manager.cc:1347] Generated new cluster ID: bc4f46ec12194200bfc67ace0fd6978c
I20260926 14:52:33.194394 26164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:52:33.196132 26171 mvcc.cc:205] Tried to move back new op lower bound from 7333619110633148416 to 7333619110609920000. Current Snapshot: MvccSnapshot[applied={T|T < 7333619110633148416}]
I20260926 14:52:33.202456 26161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a18302afea342168d4c86d1cd49b5b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db744b44910d436eb8d68d2987ba26e5" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 43649 } } peers { permanent_uuid: "3a18302afea342168d4c86d1cd49b5b1" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 33395 } } peers { permanent_uuid: "177d893e669d49e0bc1654adb0b1c961" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 39107 } } }
I20260926 14:52:33.202562 26161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1 [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:33.206355 26164 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:52:33.207484 26164 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:52:33.225811 26164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a18302afea342168d4c86d1cd49b5b1: Generated new TSK 0
I20260926 14:52:33.226768 26164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:52:34.150005 26132 catalog_manager.cc:1259] Loaded cluster ID: bc4f46ec12194200bfc67ace0fd6978c
I20260926 14:52:34.150100 26132 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961: loading cluster ID for follower catalog manager: success
I20260926 14:52:34.151793 26132 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961: acquiring CA information for follower catalog manager: success
I20260926 14:52:34.152956 26132 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 177d893e669d49e0bc1654adb0b1c961: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:52:34.189559 26159 catalog_manager.cc:1259] Loaded cluster ID: bc4f46ec12194200bfc67ace0fd6978c
I20260926 14:52:34.189639 26159 catalog_manager.cc:1552] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5: loading cluster ID for follower catalog manager: success
I20260926 14:52:34.191282 26159 catalog_manager.cc:1574] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5: acquiring CA information for follower catalog manager: success
I20260926 14:52:34.192431 26159 catalog_manager.cc:1602] T 00000000000000000000000000000000 P db744b44910d436eb8d68d2987ba26e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:52:34.302621 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 25888
I20260926 14:52:34.308239 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 25963
I20260926 14:52:34.313014 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 26037
2026-09-26T14:52:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2862 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:52:34 dist-test-slave-24mn krb5kdc[26182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:52:34 dist-test-slave-24mn krb5kdc[26182](info): set up 2 sockets
Sep 26 14:52:34 dist-test-slave-24mn krb5kdc[26182](info): commencing operation
krb5kdc: starting...
W20260926 14:52:35.287671 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:52:35 dist-test-slave-24mn krb5kdc[26182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:52:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:52:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:52:35.394941 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:41839
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:35653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:41839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:35.496569 26195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:35.496807 26195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:35.496858 26195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:35.500739 26195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:52:35.500785 26195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:35.500811 26195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:35.500831 26195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:52:35.500851 26195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:52:35.505851 26195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:35653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:41839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:41839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:35.506778 26195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:35.507699 26195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:35.515746 26203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:35.515780 26200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:35.515785 26201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:35.515904 26195 server_base.cc:1034] running on GCE node
I20260926 14:52:35.516639 26195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:35.517628 26195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:35.518784 26195 hybrid_clock.cc:648] HybridClock initialized: now 1790434355518768 us; error 33 us; skew 500 ppm
Sep 26 14:52:35 dist-test-slave-24mn krb5kdc[26182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434355, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:35.526237 26195 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:52:35.527031 26195 webserver.cc:460] Webserver started at http://127.12.68.254:42689/ using document root <none> and password file <none>
I20260926 14:52:35.527532 26195 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:35.527595 26195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:35.527783 26195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:52:35.529762 26195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "3deaec1fad4a4933bbcb3ece53e6993f"
format_stamp: "Formatted at 2026-09-26 14:52:35 on dist-test-slave-24mn"
server_key: "8c75ee50bcd2e5a504556fa7b320fe8b"
server_key_iv: "e037a9cd85eb62f25d9642eaabd11ae4"
server_key_version: "encryptionkey@0"
I20260926 14:52:35.530280 26195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "3deaec1fad4a4933bbcb3ece53e6993f"
format_stamp: "Formatted at 2026-09-26 14:52:35 on dist-test-slave-24mn"
server_key: "8c75ee50bcd2e5a504556fa7b320fe8b"
server_key_iv: "e037a9cd85eb62f25d9642eaabd11ae4"
server_key_version: "encryptionkey@0"
I20260926 14:52:35.533711 26195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:52:35.535830 26210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:35.536947 26195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:52:35.537052 26195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "3deaec1fad4a4933bbcb3ece53e6993f"
format_stamp: "Formatted at 2026-09-26 14:52:35 on dist-test-slave-24mn"
server_key: "8c75ee50bcd2e5a504556fa7b320fe8b"
server_key_iv: "e037a9cd85eb62f25d9642eaabd11ae4"
server_key_version: "encryptionkey@0"
I20260926 14:52:35.537137 26195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:35.547535 26195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:35.550246 26195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:35.550398 26195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:35.556459 26195 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:41839
I20260926 14:52:35.556479 26262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:41839 every 8 connection(s)
I20260926 14:52:35.557262 26195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:52:35.559515 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 26195
I20260926 14:52:35.559603 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:52:35.559753 26263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:35.559864 12563 external_mini_cluster.cc:1428] Setting key a65fc47a96f8cf8f2e7f458d990ad4a1
I20260926 14:52:35.566258 26263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f: Bootstrap starting.
Sep 26 14:52:35 dist-test-slave-24mn krb5kdc[26182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:35.568346 26263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:35.569093 26263 log.cc:824] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:35.571075 26263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f: No bootstrap required, opened a new log
I20260926 14:52:35.573376 26266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:35.561101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.12.68.254:41839)
0926 14:52:35.561486 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:35.561503 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:35.561542 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:52:35.562197 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:35.562219 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:35.562624 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:35.562963 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:35.563569 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:35.564980 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:35.565764 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:35.566404 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:35.566555 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:35.568715 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:35.568721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:35.568735 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:35.568764 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:35.570586 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:35.570972 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:35.570977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:35.570982 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:35.571109 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:35.571383 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:35.571386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:35.571388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:35.571845 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:35.572043 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:35.572218 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 14:52:35.572492 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260926 14:52:35.575155 26263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deaec1fad4a4933bbcb3ece53e6993f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41839 } }
I20260926 14:52:35.575378 26263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:35.575436 26263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3deaec1fad4a4933bbcb3ece53e6993f, State: Initialized, Role: FOLLOWER
I20260926 14:52:35.575811 26263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deaec1fad4a4933bbcb3ece53e6993f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41839 } }
I20260926 14:52:35.575956 26263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:52:35.575958 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:52:35.576017 26263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:52:35.576110 26263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:35.576843 26263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deaec1fad4a4933bbcb3ece53e6993f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41839 } }
I20260926 14:52:35.577140 26263 leader_election.cc:302] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3deaec1fad4a4933bbcb3ece53e6993f; no voters:
I20260926 14:52:35.577428 26263 leader_election.cc:288] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:52:35.577513 26268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:52:35.577666 26268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [term 1 LEADER]: Becoming Leader. State: Replica: 3deaec1fad4a4933bbcb3ece53e6993f, State: Running, Role: LEADER
I20260926 14:52:35.577996 26268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deaec1fad4a4933bbcb3ece53e6993f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41839 } }
I20260926 14:52:35.578477 26263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:52:35.579422 26270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3deaec1fad4a4933bbcb3ece53e6993f. Latest consensus state: current_term: 1 leader_uuid: "3deaec1fad4a4933bbcb3ece53e6993f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deaec1fad4a4933bbcb3ece53e6993f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41839 } } }
I20260926 14:52:35.579557 26270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:35.579818 26269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3deaec1fad4a4933bbcb3ece53e6993f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deaec1fad4a4933bbcb3ece53e6993f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 41839 } } }
I20260926 14:52:35.579918 26269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:35.579952 26277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:52:35.582698 26277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:52:35.587767 26277 catalog_manager.cc:1347] Generated new cluster ID: a984e986113f4e9eb117361234b6e1c8
I20260926 14:52:35.587824 26277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:52:35.610024 26277 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:52:35.611217 26277 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:52:35.619755 26277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3deaec1fad4a4933bbcb3ece53e6993f: Generated new TSK 0
I20260926 14:52:35.620510 26277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:52:35.631807 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 26195
2026-09-26T14:52:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1317 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:52:35 dist-test-slave-24mn krb5kdc[26290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:52:35 dist-test-slave-24mn krb5kdc[26290](info): set up 2 sockets
Sep 26 14:52:35 dist-test-slave-24mn krb5kdc[26290](info): commencing operation
krb5kdc: starting...
W20260926 14:52:36.567720 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:52:36 dist-test-slave-24mn krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:52:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:52:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:52:36.672576 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:40155
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:36.774923 26303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:36.775153 26303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:36.775205 26303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:36.779120 26303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:52:36.779171 26303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:36.779197 26303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:36.779219 26303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:52:36.779244 26303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:52:36.784231 26303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:40155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:36.785126 26303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:36.786026 26303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:36.794011 26308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:36.794129 26311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:36.794011 26309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:36.794245 26303 server_base.cc:1034] running on GCE node
I20260926 14:52:36.795007 26303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:36.796310 26303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:36.797477 26303 hybrid_clock.cc:648] HybridClock initialized: now 1790434356797444 us; error 46 us; skew 500 ppm
Sep 26 14:52:36 dist-test-slave-24mn krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434356, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:36.800518 26303 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:52:36.801330 26303 webserver.cc:460] Webserver started at http://127.12.68.254:43661/ using document root <none> and password file <none>
I20260926 14:52:36.801792 26303 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:36.801844 26303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:36.802007 26303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:52:36.803825 26303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "220004e75e75442ba048574197d8c6df"
format_stamp: "Formatted at 2026-09-26 14:52:36 on dist-test-slave-24mn"
server_key: "0c2c818eb1d2de7ea1b2487f3653d307"
server_key_iv: "6e63b7e282652756ac59fdb129f77721"
server_key_version: "encryptionkey@0"
I20260926 14:52:36.804349 26303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "220004e75e75442ba048574197d8c6df"
format_stamp: "Formatted at 2026-09-26 14:52:36 on dist-test-slave-24mn"
server_key: "0c2c818eb1d2de7ea1b2487f3653d307"
server_key_iv: "6e63b7e282652756ac59fdb129f77721"
server_key_version: "encryptionkey@0"
I20260926 14:52:36.807585 26303 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:52:36.809724 26318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:36.810883 26303 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:52:36.810986 26303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "220004e75e75442ba048574197d8c6df"
format_stamp: "Formatted at 2026-09-26 14:52:36 on dist-test-slave-24mn"
server_key: "0c2c818eb1d2de7ea1b2487f3653d307"
server_key_iv: "6e63b7e282652756ac59fdb129f77721"
server_key_version: "encryptionkey@0"
I20260926 14:52:36.811082 26303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:36.824774 26303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:36.827476 26303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:36.827625 26303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:36.833675 26303 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:40155
I20260926 14:52:36.833930 26370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:40155 every 8 connection(s)
I20260926 14:52:36.834502 26303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:52:36.836923 26371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:36.837146 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 26303
I20260926 14:52:36.837246 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:52:36.837515 12563 external_mini_cluster.cc:1428] Setting key 2606aba49bf8f4548b9862551c79f92d
I20260926 14:52:36.842931 26371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Bootstrap starting.
I20260926 14:52:36.845367 26371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Neither blocks nor log segments found. Creating new log.
Sep 26 14:52:36 dist-test-slave-24mn krb5kdc[26290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:36.846038 26371 log.cc:824] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:36.847612 26371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: No bootstrap required, opened a new log
I20260926 14:52:36.850248 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:36.838647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.12.68.254:40155)
0926 14:52:36.839009 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:36.839022 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:36.839055 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:52:36.839711 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:36.839726 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:36.840074 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:36.840331 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:36.841171 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:36.842141 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:36.843247 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:36.843686 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:36.843795 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:36.846285 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:36.846289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:36.846301 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:36.846327 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:36.847773 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:36.848173 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:36.848179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:36.848183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:36.848283 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:36.848527 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:36.848530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:36.848532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:36.848882 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:36.849049 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:36.849305 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 14:52:36.849594 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":97,"threads_started":1}
I20260926 14:52:36.850944 26371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } }
I20260926 14:52:36.851136 26371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:36.851190 26371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220004e75e75442ba048574197d8c6df, State: Initialized, Role: FOLLOWER
I20260926 14:52:36.851560 26371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } }
I20260926 14:52:36.851670 26371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:52:36.851725 26371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:52:36.851806 26371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:36.852659 26371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } }
I20260926 14:52:36.852949 26371 leader_election.cc:302] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 220004e75e75442ba048574197d8c6df; no voters:
I20260926 14:52:36.853196 26371 leader_election.cc:288] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:52:36.853302 26376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:52:36.853492 26376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 1 LEADER]: Becoming Leader. State: Replica: 220004e75e75442ba048574197d8c6df, State: Running, Role: LEADER
I20260926 14:52:36.853799 26376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } }
I20260926 14:52:36.854271 26371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:52:36.855273 26378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 220004e75e75442ba048574197d8c6df. Latest consensus state: current_term: 1 leader_uuid: "220004e75e75442ba048574197d8c6df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } } }
I20260926 14:52:36.855315 26377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "220004e75e75442ba048574197d8c6df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } } }
I20260926 14:52:36.855446 26378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:36.855453 26377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:36.855823 26385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:52:36.858704 26385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:52:36.864321 26385 catalog_manager.cc:1347] Generated new cluster ID: 51e68dea7220441c96ec8a699c820187
I20260926 14:52:36.864383 26385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:52:36.875712 26385 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:52:36.876562 26385 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:52:36.883265 26385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Generated new TSK 0
I20260926 14:52:36.883797 26385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:52:36.910145 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:40155
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:37.012194 26397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:37.012420 26397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:37.012485 26397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:37.016460 26397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:37.016518 26397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:37.016594 26397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:52:37.021572 26397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:37.022516 26397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:37.023437 26397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:37.032347 26405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:37.032377 26402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:37.032377 26403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:37.032557 26397 server_base.cc:1034] running on GCE node
I20260926 14:52:37.032969 26397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:37.033568 26397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:37.034713 26397 hybrid_clock.cc:648] HybridClock initialized: now 1790434357034684 us; error 42 us; skew 500 ppm
Sep 26 14:52:37 dist-test-slave-24mn krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434357, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:37.037143 26397 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:52:37.037956 26397 webserver.cc:460] Webserver started at http://127.12.68.193:34983/ using document root <none> and password file <none>
I20260926 14:52:37.038455 26397 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:37.038522 26397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:37.038718 26397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:52:37.040674 26397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "b1490811f65a45b4a3ba202047b1bd1e"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "c9001a600b3ce1fde426681d5baf6d40"
server_key_iv: "a1b0b743b362324148ae0a82fae0f14b"
server_key_version: "encryptionkey@0"
I20260926 14:52:37.041167 26397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "b1490811f65a45b4a3ba202047b1bd1e"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "c9001a600b3ce1fde426681d5baf6d40"
server_key_iv: "a1b0b743b362324148ae0a82fae0f14b"
server_key_version: "encryptionkey@0"
I20260926 14:52:37.044508 26397 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:52:37.046607 26412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:37.047703 26397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:52:37.047808 26397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "b1490811f65a45b4a3ba202047b1bd1e"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "c9001a600b3ce1fde426681d5baf6d40"
server_key_iv: "a1b0b743b362324148ae0a82fae0f14b"
server_key_version: "encryptionkey@0"
I20260926 14:52:37.047926 26397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:37.062520 26397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:37.065161 26397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:37.065322 26397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:37.065858 26397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:37.066606 26397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:37.066650 26397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:37.066700 26397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:37.066728 26397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:37.074921 26397 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:43161
I20260926 14:52:37.074967 26525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:43161 every 8 connection(s)
I20260926 14:52:37.075737 26397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:52:37 dist-test-slave-24mn krb5kdc[26290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434357, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:37.085327 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 26397
I20260926 14:52:37.085457 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:52:37.085767 12563 external_mini_cluster.cc:1428] Setting key e32a304a2116cbd7ce0c42377185476a
I20260926 14:52:37.088490 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.077340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:39323 (local address 127.12.68.254:40155)
0926 14:52:37.077465 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:37.077468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:37.078244 (+ 776us) server_negotiation.cc:407] Connection header received
0926 14:52:37.079486 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:37.079488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:37.079533 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:37.079600 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:37.080732 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.081255 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.082278 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.082496 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.082562 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.084740 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:37.084743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:37.084745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:37.084766 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:37.086289 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:37.086785 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:37.086788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:37.086789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:37.086887 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:37.087396 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:37.087400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:37.087402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:37.087593 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:37.087678 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:37.088190 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 14:52:37.088316 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:52:37.089308 26528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.077694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:39323)
0926 14:52:37.078104 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:37.078144 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:37.079302 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:37.079746 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:37.079752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:37.080159 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:37.080614 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.080627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.081342 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.081345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.082194 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.082202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.082597 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.082599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.082731 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.083243 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:37.083258 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:37.084593 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:37.086387 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:37.086391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:37.086402 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:37.086641 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:37.087172 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:37.087175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:37.087176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:37.087306 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:37.087688 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:37.087694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:37.087960 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 14:52:37.088137 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 14:52:37.088444 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":105,"threads_started":1}
I20260926 14:52:37.090418 26526 heartbeater.cc:347] Connected to a master server at 127.12.68.254:40155
I20260926 14:52:37.090639 26526 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:37.091094 26526 heartbeater.cc:511] Master 127.12.68.254:40155 requested a full tablet report, sending...
I20260926 14:52:37.092458 26335 ts_manager.cc:194] Registered new tserver with Master: b1490811f65a45b4a3ba202047b1bd1e (127.12.68.193:43161)
I20260926 14:52:37.093539 26335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:39323
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:52:37.110496 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:40155
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:37.218140 26531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:37.218394 26531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:37.218437 26531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:37.222401 26531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:37.222460 26531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:37.222517 26531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:52:37.227766 26531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:37.228802 26531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:37.229787 26531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:37.238850 26536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:37.238883 26539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:37.238883 26537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:37.238967 26531 server_base.cc:1034] running on GCE node
I20260926 14:52:37.239504 26531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:37.240202 26531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:37.243459 26531 hybrid_clock.cc:648] HybridClock initialized: now 1790434357243439 us; error 37 us; skew 500 ppm
Sep 26 14:52:37 dist-test-slave-24mn krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434357, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:37.246059 26531 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:52:37.246937 26531 webserver.cc:460] Webserver started at http://127.12.68.194:39003/ using document root <none> and password file <none>
I20260926 14:52:37.247463 26531 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:37.247526 26531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:37.247704 26531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:52:37.249699 26531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "4c6e58cd584d4fe6ab89bf89b508fe43"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "4323cb332f36b2c1f5cdb6bb152eeca3"
server_key_iv: "56715596659495e25bb391a51520a7d8"
server_key_version: "encryptionkey@0"
I20260926 14:52:37.250233 26531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "4c6e58cd584d4fe6ab89bf89b508fe43"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "4323cb332f36b2c1f5cdb6bb152eeca3"
server_key_iv: "56715596659495e25bb391a51520a7d8"
server_key_version: "encryptionkey@0"
I20260926 14:52:37.253834 26531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:52:37.256346 26546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:37.257501 26531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:52:37.257601 26531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "4c6e58cd584d4fe6ab89bf89b508fe43"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "4323cb332f36b2c1f5cdb6bb152eeca3"
server_key_iv: "56715596659495e25bb391a51520a7d8"
server_key_version: "encryptionkey@0"
I20260926 14:52:37.257690 26531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:37.278133 26531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:37.281083 26531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:37.281256 26531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:37.281813 26531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:37.282604 26531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:37.282658 26531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:37.282711 26531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:37.282734 26531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:37.290817 26531 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:36789
I20260926 14:52:37.290912 26659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:36789 every 8 connection(s)
I20260926 14:52:37.291661 26531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:52:37.296307 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 26531
I20260926 14:52:37.296419 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:52:37.296667 12563 external_mini_cluster.cc:1428] Setting key 6909e119051c98ebdfe79c913f04c689
Sep 26 14:52:37 dist-test-slave-24mn krb5kdc[26290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434357, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:37.305055 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.293382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:53785 (local address 127.12.68.254:40155)
0926 14:52:37.293558 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:37.293563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:37.294333 (+ 770us) server_negotiation.cc:407] Connection header received
0926 14:52:37.295592 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:37.295595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:37.295640 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:37.295709 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:37.296916 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.297515 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.298648 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.298857 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.298940 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.301666 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:37.301669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:37.301671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:37.301695 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:37.302995 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:37.303569 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:37.303572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:37.303573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:37.303637 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:37.304000 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:37.304003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:37.304004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:37.304172 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:37.304239 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:37.304831 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 14:52:37.304936 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
I20260926 14:52:37.306090 26662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.293762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:53785)
0926 14:52:37.294191 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:37.294232 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:37.295390 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:37.295855 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:37.295862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:37.296245 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:37.296784 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.296799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.297630 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.297633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.298539 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.298549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.298983 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.298988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.299168 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.299744 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:37.299764 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:37.301463 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:37.303119 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:37.303125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:37.303140 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:37.303471 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:37.303711 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:37.303714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:37.303716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:37.303918 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:37.304265 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:37.304271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:37.304596 (+ 325us) client_negotiation.cc:769] Sending connection context
0926 14:52:37.304805 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 14:52:37.305148 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":119,"threads_started":1}
I20260926 14:52:37.307152 26660 heartbeater.cc:347] Connected to a master server at 127.12.68.254:40155
I20260926 14:52:37.307397 26660 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
I20260926 14:52:37.307920 26660 heartbeater.cc:511] Master 127.12.68.254:40155 requested a full tablet report, sending...
I20260926 14:52:37.310200 26335 ts_manager.cc:194] Registered new tserver with Master: 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789)
I20260926 14:52:37.310746 26335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:53785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:52:37.322829 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:40155
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:37.427623 26665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:37.427834 26665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:37.427871 26665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:37.431811 26665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:37.431860 26665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:37.431932 26665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:52:37.436870 26665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:37.437783 26665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:37.438692 26665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:37.447466 26671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:37.447484 26670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:37.447552 26673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:37.447683 26665 server_base.cc:1034] running on GCE node
I20260926 14:52:37.448071 26665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:37.448617 26665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:37.449755 26665 hybrid_clock.cc:648] HybridClock initialized: now 1790434357449737 us; error 29 us; skew 500 ppm
Sep 26 14:52:37 dist-test-slave-24mn krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434357, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:37.452302 26665 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:52:37.453016 26665 webserver.cc:460] Webserver started at http://127.12.68.195:35075/ using document root <none> and password file <none>
I20260926 14:52:37.453480 26665 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:37.453531 26665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:37.453682 26665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:52:37.455591 26665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "7d918660f51146b78e50a4ce427a31e7"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "ec8e4aeba2141f85e554a8f51de55c4b"
server_key_iv: "23ff4971f75fa7ca892bf699f279bdea"
server_key_version: "encryptionkey@0"
I20260926 14:52:37.456096 26665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "7d918660f51146b78e50a4ce427a31e7"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "ec8e4aeba2141f85e554a8f51de55c4b"
server_key_iv: "23ff4971f75fa7ca892bf699f279bdea"
server_key_version: "encryptionkey@0"
I20260926 14:52:37.459318 26665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 14:52:37.461472 26680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:37.462569 26665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:52:37.462664 26665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "7d918660f51146b78e50a4ce427a31e7"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "ec8e4aeba2141f85e554a8f51de55c4b"
server_key_iv: "23ff4971f75fa7ca892bf699f279bdea"
server_key_version: "encryptionkey@0"
I20260926 14:52:37.462749 26665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:37.474123 26665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:37.476778 26665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:37.476929 26665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:37.477406 26665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:37.478101 26665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:37.478142 26665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:37.478189 26665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:37.478219 26665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:37.485797 26665 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:44993
I20260926 14:52:37.485874 26793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:44993 every 8 connection(s)
I20260926 14:52:37.486559 26665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:52:37.488193 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 26665
I20260926 14:52:37.488302 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:52:37.488543 12563 external_mini_cluster.cc:1428] Setting key c6a460c1883e35afcf7e82df37cf7661
Sep 26 14:52:37 dist-test-slave-24mn krb5kdc[26290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434357, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:37.498234 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.488177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:46833 (local address 127.12.68.254:40155)
0926 14:52:37.488327 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:37.488331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:37.489079 (+ 748us) server_negotiation.cc:407] Connection header received
0926 14:52:37.490340 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:37.490343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:37.490405 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:37.490472 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:37.491496 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.492055 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.493101 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.493318 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.493407 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.495481 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:37.495484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:37.495486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:37.495508 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:37.496759 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:37.497148 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:37.497150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:37.497151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:37.497216 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:37.497451 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:37.497453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:37.497454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:37.497613 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:37.497679 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:37.498038 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 14:52:37.498123 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 14:52:37.498911 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.488493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:46833)
0926 14:52:37.488939 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:37.488986 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:37.490155 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:37.490637 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:37.490643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:37.490958 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:37.491373 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.491385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.492156 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.492159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.492996 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.493004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.493435 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.493439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.493589 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.494035 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:37.494049 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:37.495331 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:37.496845 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:37.496850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:37.496861 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:37.497074 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:37.497278 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:37.497280 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:37.497281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:37.497393 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:37.497677 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:37.497681 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:37.497873 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 14:52:37.498006 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 14:52:37.498212 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":128,"threads_started":1}
I20260926 14:52:37.499846 26794 heartbeater.cc:347] Connected to a master server at 127.12.68.254:40155
I20260926 14:52:37.500069 26794 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:37.500453 26794 heartbeater.cc:511] Master 127.12.68.254:40155 requested a full tablet report, sending...
I20260926 14:52:37.501363 26335 ts_manager.cc:194] Registered new tserver with Master: 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993)
I20260926 14:52:37.501875 26335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:46833
I20260926 14:52:37.502449 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:52:37.510449 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.504155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.12.68.254:40155)
0926 14:52:37.504262 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:37.504266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:37.504426 (+ 160us) server_negotiation.cc:407] Connection header received
0926 14:52:37.504533 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:37.504536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:37.504574 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:37.504635 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:37.505258 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.505740 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.506515 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.506729 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.506807 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.507906 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:37.507910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:37.507913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:37.507936 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:37.508998 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:37.509352 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:37.509356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:37.509358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:37.509429 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:37.509674 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:37.509677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:37.509679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:37.509883 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:37.509995 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:37.510222 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 14:52:37.510317 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:52:37.512919 26334 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:52:37.515727 26334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:52:37.528841 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.523555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:44993 (local address 127.0.0.1:48738)
0926 14:52:37.524065 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:37.524077 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:37.524155 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:37.524712 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:37.524715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:37.524732 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:37.524829 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.524835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.526265 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.526269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.527302 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.527311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.528331 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.528335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.528436 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.528474 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 14:52:37.528592 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 14:52:37.528703 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":18,"thread_start_us":88,"threads_started":1}
I20260926 14:52:37.528846 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.523631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.12.68.195:44993)
0926 14:52:37.524329 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:37.524339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:37.524357 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:52:37.524426 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:37.524431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:37.524578 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:37.524966 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:37.525091 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.526149 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.527412 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.528213 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.528321 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.528464 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:37.528625 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 14:52:37.528715 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":87,"threads_started":1}
I20260926 14:52:37.528872 26805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.522959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:43161 (local address 127.0.0.1:49160)
0926 14:52:37.523876 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:37.523939 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:37.524077 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:37.524573 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:37.524578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:37.524600 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:37.524712 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.524718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.526111 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.526114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.527419 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.527425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.528524 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.528527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.528603 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.528625 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:52:37.528690 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:52:37.528752 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":82,"thread_start_us":66,"threads_started":1}
I20260926 14:52:37.529366 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.523093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.12.68.193:43161)
0926 14:52:37.523420 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:37.523432 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:37.524020 (+ 588us) server_negotiation.cc:407] Connection header received
0926 14:52:37.524232 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:37.524237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:37.524428 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:37.524569 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:37.524826 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.525993 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.527529 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.528279 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.528360 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.528464 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:37.529141 (+ 677us) server_negotiation.cc:299] Negotiation successful
0926 14:52:37.529235 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":140,"threads_started":1}
I20260926 14:52:37.529678 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.523095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:36789 (local address 127.0.0.1:41882)
0926 14:52:37.523866 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:37.523939 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:37.524056 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:37.524968 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:37.524971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:37.524984 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:37.525055 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.525059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.525973 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.525977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.527414 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.527425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.528863 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.528867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.529277 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.529295 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:52:37.529360 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:52:37.529563 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":94,"thread_start_us":93,"threads_started":1}
I20260926 14:52:37.531701 26728 tablet_service.cc:1467] Processing CreateTablet for tablet 493bfe58692b4866b3bf6e7296516780 (DEFAULT_TABLE table=test-table [id=871743935fe04cd7bf35f86da5017cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:52:37.531700 26460 tablet_service.cc:1467] Processing CreateTablet for tablet 493bfe58692b4866b3bf6e7296516780 (DEFAULT_TABLE table=test-table [id=871743935fe04cd7bf35f86da5017cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:52:37.532557 26460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493bfe58692b4866b3bf6e7296516780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:37.532572 26728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493bfe58692b4866b3bf6e7296516780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:37.533946 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.523697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.12.68.194:36789)
0926 14:52:37.524379 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:37.524391 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:37.524410 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:52:37.524482 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:37.524487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:37.524679 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:37.524833 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:37.525159 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.525860 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.527958 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.528602 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.528688 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.528807 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:37.529405 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 14:52:37.533809 (+ 4404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":124,"threads_started":1}
I20260926 14:52:37.536612 26594 tablet_service.cc:1467] Processing CreateTablet for tablet 493bfe58692b4866b3bf6e7296516780 (DEFAULT_TABLE table=test-table [id=871743935fe04cd7bf35f86da5017cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:52:37.537572 26594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493bfe58692b4866b3bf6e7296516780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:37.538156 26811 tablet_bootstrap.cc:492] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Bootstrap starting.
I20260926 14:52:37.539530 26812 tablet_bootstrap.cc:492] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7: Bootstrap starting.
I20260926 14:52:37.540541 26811 tablet_bootstrap.cc:654] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:37.541285 26811 log.cc:824] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:37.541944 26812 tablet_bootstrap.cc:654] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:37.542752 26812 log.cc:824] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:37.542894 26811 tablet_bootstrap.cc:492] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: No bootstrap required, opened a new log
I20260926 14:52:37.543059 26811 ts_tablet_manager.cc:1397] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:52:37.544596 26812 tablet_bootstrap.cc:492] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7: No bootstrap required, opened a new log
I20260926 14:52:37.544812 26812 ts_tablet_manager.cc:1397] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:52:37.545015 26815 tablet_bootstrap.cc:492] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43: Bootstrap starting.
I20260926 14:52:37.546795 26811 raft_consensus.cc:348] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } }
I20260926 14:52:37.547127 26811 raft_consensus.cc:374] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:37.547199 26811 raft_consensus.cc:729] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1490811f65a45b4a3ba202047b1bd1e, State: Initialized, Role: FOLLOWER
I20260926 14:52:37.547519 26815 tablet_bootstrap.cc:654] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:37.547681 26811 consensus_queue.cc:260] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } }
I20260926 14:52:37.547735 26812 raft_consensus.cc:348] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } }
I20260926 14:52:37.547976 26812 raft_consensus.cc:374] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:37.548027 26812 raft_consensus.cc:729] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d918660f51146b78e50a4ce427a31e7, State: Initialized, Role: FOLLOWER
I20260926 14:52:37.548342 26815 log.cc:824] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:37.548509 26526 heartbeater.cc:503] Master 127.12.68.254:40155 was elected leader, sending a full tablet report...
I20260926 14:52:37.548440 26812 consensus_queue.cc:260] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } }
I20260926 14:52:37.548673 26811 ts_tablet_manager.cc:1428] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:52:37.549180 26794 heartbeater.cc:503] Master 127.12.68.254:40155 was elected leader, sending a full tablet report...
I20260926 14:52:37.549324 26812 ts_tablet_manager.cc:1428] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 14:52:37.550426 26815 tablet_bootstrap.cc:492] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43: No bootstrap required, opened a new log
I20260926 14:52:37.550693 26815 ts_tablet_manager.cc:1397] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:52:37.554018 26815 raft_consensus.cc:348] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } }
I20260926 14:52:37.554216 26815 raft_consensus.cc:374] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:37.554275 26815 raft_consensus.cc:729] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6e58cd584d4fe6ab89bf89b508fe43, State: Initialized, Role: FOLLOWER
I20260926 14:52:37.554587 26815 consensus_queue.cc:260] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } }
I20260926 14:52:37.555083 26660 heartbeater.cc:503] Master 127.12.68.254:40155 was elected leader, sending a full tablet report...
I20260926 14:52:37.555349 26815 ts_tablet_manager.cc:1428] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
W20260926 14:52:37.576776 26527 tablet.cc:2369] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:52:37.595101 26816 raft_consensus.cc:482] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:52:37.595249 26816 raft_consensus.cc:504] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } }
I20260926 14:52:37.596241 26816 leader_election.cc:288] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789), 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993)
I20260926 14:52:37.600361 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.596534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:33965 (local address 127.12.68.194:36789)
0926 14:52:37.596654 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:37.596658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:37.596825 (+ 167us) server_negotiation.cc:407] Connection header received
0926 14:52:37.596958 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:37.596962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:37.597012 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:37.597085 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:37.597362 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.597879 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.599412 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.599798 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.599914 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.599966 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:37.600178 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 14:52:37.600249 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:52:37.600396 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.596447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:36789 (local address 127.12.68.193:33965)
0926 14:52:37.596726 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:37.596739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:37.596845 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:37.597099 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:37.597102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:37.597122 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:37.597234 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.597240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.597989 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.597991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.599265 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.599276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.599920 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.599924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.600050 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.600080 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 14:52:37.600142 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 14:52:37.600212 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":6,"thread_start_us":73,"threads_started":1}
I20260926 14:52:37.600389 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.596572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:43891 (local address 127.12.68.195:44993)
0926 14:52:37.596686 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:37.596690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:37.596950 (+ 260us) server_negotiation.cc:407] Connection header received
0926 14:52:37.597102 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:37.597105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:37.597151 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:37.597218 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:37.597451 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.597966 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.599391 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.599787 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.599949 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.600000 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:37.600204 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 14:52:37.600269 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:52:37.600419 26821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.596463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:44993 (local address 127.12.68.193:43891)
0926 14:52:37.596849 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:37.596860 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:37.597003 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:37.597230 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:37.597233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:37.597246 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:37.597346 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.597352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.598068 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.598071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.599265 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:37.599276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.599922 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.599926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:37.600094 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.600112 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:52:37.600175 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:52:37.600245 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":92,"thread_start_us":127,"threads_started":1}
I20260926 14:52:37.600950 26748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493bfe58692b4866b3bf6e7296516780" candidate_uuid: "b1490811f65a45b4a3ba202047b1bd1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d918660f51146b78e50a4ce427a31e7" is_pre_election: true
I20260926 14:52:37.600955 26614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493bfe58692b4866b3bf6e7296516780" candidate_uuid: "b1490811f65a45b4a3ba202047b1bd1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" is_pre_election: true
I20260926 14:52:37.601202 26614 raft_consensus.cc:2454] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1490811f65a45b4a3ba202047b1bd1e in term 0.
I20260926 14:52:37.601214 26748 raft_consensus.cc:2454] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1490811f65a45b4a3ba202047b1bd1e in term 0.
I20260926 14:52:37.601608 26413 leader_election.cc:302] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6e58cd584d4fe6ab89bf89b508fe43, b1490811f65a45b4a3ba202047b1bd1e; no voters:
I20260926 14:52:37.601845 26816 raft_consensus.cc:2781] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:52:37.601919 26816 raft_consensus.cc:482] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:52:37.601951 26816 raft_consensus.cc:3037] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:37.602959 26816 raft_consensus.cc:504] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } }
I20260926 14:52:37.603260 26816 leader_election.cc:288] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 1 election: Requested vote from peers 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789), 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993)
I20260926 14:52:37.603554 26748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493bfe58692b4866b3bf6e7296516780" candidate_uuid: "b1490811f65a45b4a3ba202047b1bd1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d918660f51146b78e50a4ce427a31e7"
I20260926 14:52:37.603547 26614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493bfe58692b4866b3bf6e7296516780" candidate_uuid: "b1490811f65a45b4a3ba202047b1bd1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43"
I20260926 14:52:37.603648 26748 raft_consensus.cc:3037] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:37.603657 26614 raft_consensus.cc:3037] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:37.604624 26748 raft_consensus.cc:2454] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1490811f65a45b4a3ba202047b1bd1e in term 1.
I20260926 14:52:37.604686 26614 raft_consensus.cc:2454] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1490811f65a45b4a3ba202047b1bd1e in term 1.
I20260926 14:52:37.604871 26414 leader_election.cc:302] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d918660f51146b78e50a4ce427a31e7, b1490811f65a45b4a3ba202047b1bd1e; no voters:
I20260926 14:52:37.605062 26816 raft_consensus.cc:2781] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:52:37.605229 26816 raft_consensus.cc:686] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 1 LEADER]: Becoming Leader. State: Replica: b1490811f65a45b4a3ba202047b1bd1e, State: Running, Role: LEADER
I20260926 14:52:37.605527 26816 consensus_queue.cc:237] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } }
I20260926 14:52:37.608206 26335 catalog_manager.cc:5495] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e reported cstate change: term changed from 0 to 1, leader changed from <none> to b1490811f65a45b4a3ba202047b1bd1e (127.12.68.193). New cstate: current_term: 1 leader_uuid: "b1490811f65a45b4a3ba202047b1bd1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } health_report { overall_health: UNKNOWN } } }
I20260926 14:52:37.622794 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.619993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49172 (local address 127.12.68.193:43161)
0926 14:52:37.620090 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:37.620093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:37.620105 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:52:37.620205 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:37.620207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:37.620254 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:37.620335 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:52:37.620506 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.620978 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.621722 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:37.621931 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:37.622002 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:37.622202 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:52:37.622558 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:52:37.622637 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:37.622663 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 14:52:37.622704 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:52:37.627687 26614 raft_consensus.cc:1261] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 1 FOLLOWER]: Refusing update from remote peer b1490811f65a45b4a3ba202047b1bd1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:52:37.627687 26748 raft_consensus.cc:1261] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7 [term 1 FOLLOWER]: Refusing update from remote peer b1490811f65a45b4a3ba202047b1bd1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:52:37.628328 26816 consensus_queue.cc:1035] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:52:37.628540 26822 consensus_queue.cc:1035] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:52:37.635044 26825 mvcc.cc:205] Tried to move back new op lower bound from 7333619128837337088 to 7333619128754438144. Current Snapshot: MvccSnapshot[applied={T|T < 7333619128837337088}]
I20260926 14:52:37.637226 26826 mvcc.cc:205] Tried to move back new op lower bound from 7333619128837337088 to 7333619128754438144. Current Snapshot: MvccSnapshot[applied={T|T < 7333619128837337088}]
I20260926 14:52:37.637595 26827 mvcc.cc:205] Tried to move back new op lower bound from 7333619128837337088 to 7333619128754438144. Current Snapshot: MvccSnapshot[applied={T|T < 7333619128837337088}]
I20260926 14:52:37.643157 26335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:52:37.643309 26335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:52:37.645496 26335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Sending DeleteTablet for 3 replicas of tablet 493bfe58692b4866b3bf6e7296516780
I20260926 14:52:37.646143 26460 tablet_service.cc:1514] Processing DeleteTablet for tablet 493bfe58692b4866b3bf6e7296516780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:52:37 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:49160
I20260926 14:52:37.646157 26594 tablet_service.cc:1514] Processing DeleteTablet for tablet 493bfe58692b4866b3bf6e7296516780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:52:37 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:41882
I20260926 14:52:37.646319 26728 tablet_service.cc:1514] Processing DeleteTablet for tablet 493bfe58692b4866b3bf6e7296516780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:52:37 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:48738
I20260926 14:52:37.646728 26837 tablet_replica.cc:330] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7: stopping tablet replica
I20260926 14:52:37.646744 26836 tablet_replica.cc:330] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: stopping tablet replica
I20260926 14:52:37.646756 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 26397
I20260926 14:52:37.647001 26836 raft_consensus.cc:2229] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:52:37.647042 26837 raft_consensus.cc:2229] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:52:37.647349 26835 tablet_replica.cc:330] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43: stopping tablet replica
I20260926 14:52:37.647449 26837 raft_consensus.cc:2258] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:52:37.647572 26835 raft_consensus.cc:2229] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:52:37.647819 26835 raft_consensus.cc:2258] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:52:37.648908 26837 ts_tablet_manager.cc:1905] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:52:37.649142 26835 ts_tablet_manager.cc:1905] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:52:37.651028 26837 ts_tablet_manager.cc:1918] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:52:37.651108 26837 log.cc:1196] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/493bfe58692b4866b3bf6e7296516780
I20260926 14:52:37.651384 26837 ts_tablet_manager.cc:1939] T 493bfe58692b4866b3bf6e7296516780 P 7d918660f51146b78e50a4ce427a31e7: Deleting consensus metadata
I20260926 14:52:37.651531 26835 ts_tablet_manager.cc:1918] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:52:37.651618 26835 log.cc:1196] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/493bfe58692b4866b3bf6e7296516780
I20260926 14:52:37.651918 26835 ts_tablet_manager.cc:1939] T 493bfe58692b4866b3bf6e7296516780 P 4c6e58cd584d4fe6ab89bf89b508fe43: Deleting consensus metadata
I20260926 14:52:37.652390 26320 catalog_manager.cc:4841] TS 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993): tablet 493bfe58692b4866b3bf6e7296516780 (table test-table [id=871743935fe04cd7bf35f86da5017cf9]) successfully deleted
I20260926 14:52:37.652865 26319 catalog_manager.cc:4841] TS 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789): tablet 493bfe58692b4866b3bf6e7296516780 (table test-table [id=871743935fe04cd7bf35f86da5017cf9]) successfully deleted
W20260926 14:52:37.653714 26321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:43161 (error 108)
I20260926 14:52:37.653894 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 26531
I20260926 14:52:37.654870 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:37.654613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:43161 (local address 127.0.0.1:49176)
0926 14:52:37.654729 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:37.654785 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:43161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 14:52:37.655006 26321 catalog_manager.cc:4584] TS b1490811f65a45b4a3ba202047b1bd1e (127.12.68.193:43161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 493bfe58692b4866b3bf6e7296516780: Network error: Client connection negotiation failed: client connection to 127.12.68.193:43161: connect: Connection refused (error 111)
I20260926 14:52:37.658759 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 26665
I20260926 14:52:37.663385 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 26303
I20260926 14:52:37.668488 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:40155
--webserver_interface=127.12.68.254
--webserver_port=43661
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:37.770432 26839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:37.770653 26839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:37.770702 26839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:37.774572 26839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:52:37.774619 26839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:37.774645 26839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:37.774667 26839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:52:37.774694 26839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:52:37.779608 26839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:40155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=43661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:37.780534 26839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:37.781463 26839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:37.792992 26848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:37.793016 26846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:37.793027 26845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:37.793090 26839 server_base.cc:1034] running on GCE node
I20260926 14:52:37.793798 26839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:37.794799 26839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:37.795954 26839 hybrid_clock.cc:648] HybridClock initialized: now 1790434357795927 us; error 31 us; skew 500 ppm
Sep 26 14:52:37 dist-test-slave-24mn krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434357, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:37.798401 26839 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:52:37.799178 26839 webserver.cc:460] Webserver started at http://127.12.68.254:43661/ using document root <none> and password file <none>
I20260926 14:52:37.799681 26839 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:37.799752 26839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:37.803313 26839 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:52:37.805367 26855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:37.806488 26839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:52:37.806653 26839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "220004e75e75442ba048574197d8c6df"
format_stamp: "Formatted at 2026-09-26 14:52:36 on dist-test-slave-24mn"
server_key: "0c2c818eb1d2de7ea1b2487f3653d307"
server_key_iv: "6e63b7e282652756ac59fdb129f77721"
server_key_version: "encryptionkey@0"
I20260926 14:52:37.807199 26839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:37.826123 26839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:37.828689 26839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:37.828848 26839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:37.835216 26839 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:40155
I20260926 14:52:37.835459 26907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:40155 every 8 connection(s)
I20260926 14:52:37.836226 26839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:52:37.840245 26908 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:52:37.841751 26908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Bootstrap starting.
I20260926 14:52:37.843506 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 26839
I20260926 14:52:37.843940 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:43161
--local_ip_for_outbound_sockets=127.12.68.193
--tserver_master_addrs=127.12.68.254:40155
--webserver_port=34983
--webserver_interface=127.12.68.193
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:52:37.858098 26908 log.cc:824] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:37.866950 26908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:52:37.867249 26908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Bootstrap complete.
I20260926 14:52:37.870945 26908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } }
I20260926 14:52:37.871312 26908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220004e75e75442ba048574197d8c6df, State: Initialized, Role: FOLLOWER
I20260926 14:52:37.871759 26908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } }
I20260926 14:52:37.871893 26908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:52:37.871968 26908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:52:37.872066 26908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:52:37.873740 26908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } }
I20260926 14:52:37.874038 26908 leader_election.cc:302] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 220004e75e75442ba048574197d8c6df; no voters:
I20260926 14:52:37.874293 26908 leader_election.cc:288] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:52:37.874398 26912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:52:37.874607 26912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 2 LEADER]: Becoming Leader. State: Replica: 220004e75e75442ba048574197d8c6df, State: Running, Role: LEADER
I20260926 14:52:37.874871 26912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } }
I20260926 14:52:37.875422 26908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:52:37.875744 26912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 220004e75e75442ba048574197d8c6df. Latest consensus state: current_term: 2 leader_uuid: "220004e75e75442ba048574197d8c6df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } } }
I20260926 14:52:37.875845 26912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:37.876103 26913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "220004e75e75442ba048574197d8c6df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } } }
I20260926 14:52:37.876186 26916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:52:37.876196 26913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:37.880532 26916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:52:37.881108 26916 catalog_manager.cc:1259] Loaded cluster ID: 51e68dea7220441c96ec8a699c820187
I20260926 14:52:37.881166 26916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:52:37.882494 26916 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 14:52:37.887269 26916 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 14:52:37.892057 26916 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:52:37.892724 26916 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Loaded TSK: 0
I20260926 14:52:37.893821 26916 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:52:37.952026 26910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:37.952246 26910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:37.952302 26910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:37.956197 26910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:37.956243 26910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:37.956321 26910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:52:37.961356 26910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:43161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=34983
--tserver_master_addrs=127.12.68.254:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:37.962248 26910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:37.963152 26910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:37.972970 26936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:37.973114 26910 server_base.cc:1034] running on GCE node
W20260926 14:52:37.972985 26935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:37.973218 26938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:37.973562 26910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:37.974153 26910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:37.975317 26910 hybrid_clock.cc:648] HybridClock initialized: now 1790434357975291 us; error 40 us; skew 500 ppm
Sep 26 14:52:37 dist-test-slave-24mn krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434357, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:37.978068 26910 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:52:37.979058 26910 webserver.cc:460] Webserver started at http://127.12.68.193:34983/ using document root <none> and password file <none>
I20260926 14:52:37.979725 26910 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:37.979835 26910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:37.984563 26910 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:52:37.987011 26945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:37.988616 26910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:52:37.988734 26910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "b1490811f65a45b4a3ba202047b1bd1e"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "c9001a600b3ce1fde426681d5baf6d40"
server_key_iv: "a1b0b743b362324148ae0a82fae0f14b"
server_key_version: "encryptionkey@0"
I20260926 14:52:37.989387 26910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:38.006652 26910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:38.009881 26910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:38.010068 26910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:38.010735 26910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:38.012017 26952 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:52:38.015769 26910 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:52:38.015834 26910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 14:52:38.015913 26910 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:52:38.017823 26910 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:52:38.017872 26910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 14:52:38.018159 26952 tablet_bootstrap.cc:492] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Bootstrap starting.
I20260926 14:52:38.027077 26910 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:43161
I20260926 14:52:38.027130 27059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:43161 every 8 connection(s)
I20260926 14:52:38.027944 26910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:52:38.029816 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 26910
I20260926 14:52:38.030314 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:36789
--local_ip_for_outbound_sockets=127.12.68.194
--tserver_master_addrs=127.12.68.254:40155
--webserver_port=39003
--webserver_interface=127.12.68.194
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:52:38.038020 26952 log.cc:824] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:52:38 dist-test-slave-24mn krb5kdc[26290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434357, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:38.043962 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.029911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:59203 (local address 127.12.68.254:40155)
0926 14:52:38.030627 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:38.030647 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:38.031050 (+ 403us) server_negotiation.cc:407] Connection header received
0926 14:52:38.032223 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:38.032244 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:38.032691 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:38.033057 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:38.034018 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.034985 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.036138 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.036814 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.036963 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.039534 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:38.039540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:38.039555 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:38.039583 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:38.041109 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:38.041566 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:38.041572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:38.041575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:38.041683 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:38.041970 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:38.041973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:38.041975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:38.042295 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:38.042408 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:38.042894 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 14:52:38.043130 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":128,"threads_started":1}
I20260926 14:52:38.043980 27063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.029923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:59203)
0926 14:52:38.030899 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:38.030937 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:38.031677 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:38.033110 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:38.033116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:38.033427 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:38.033864 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.033878 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.035120 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.035123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.035997 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.036007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.037171 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.037175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.037338 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.037844 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:38.037860 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:38.039351 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:38.041217 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:38.041223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:38.041235 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:38.041465 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:38.041765 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:38.041768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:38.041769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:38.041889 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:38.042398 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:38.042404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:38.042620 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 14:52:38.042772 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 14:52:38.043085 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":115,"threads_started":1}
I20260926 14:52:38.046080 27060 heartbeater.cc:347] Connected to a master server at 127.12.68.254:40155
I20260926 14:52:38.046255 26952 tablet_bootstrap.cc:492] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:52:38.046277 27060 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:38.046617 26952 tablet_bootstrap.cc:492] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Bootstrap complete.
I20260926 14:52:38.046725 27060 heartbeater.cc:511] Master 127.12.68.254:40155 requested a full tablet report, sending...
I20260926 14:52:38.047930 26952 ts_tablet_manager.cc:1397] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Time spent bootstrapping tablet: real 0.030s user 0.020s sys 0.010s
I20260926 14:52:38.048807 26872 ts_manager.cc:194] Registered new tserver with Master: b1490811f65a45b4a3ba202047b1bd1e (127.12.68.193:43161)
I20260926 14:52:38.050639 26952 raft_consensus.cc:348] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } }
I20260926 14:52:38.051131 26952 raft_consensus.cc:729] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1490811f65a45b4a3ba202047b1bd1e, State: Initialized, Role: FOLLOWER
I20260926 14:52:38.051612 26952 consensus_queue.cc:260] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } }
I20260926 14:52:38.051916 26872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:59203
I20260926 14:52:38.052538 26952 ts_tablet_manager.cc:1428] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:52:38.054189 27060 heartbeater.cc:503] Master 127.12.68.254:40155 was elected leader, sending a full tablet report...
I20260926 14:52:38.057230 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.051979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.12.68.193:43161)
0926 14:52:38.052427 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:38.052437 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:38.053211 (+ 774us) server_negotiation.cc:407] Connection header received
0926 14:52:38.053365 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:38.053370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:38.053551 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:38.053691 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:38.053967 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.054863 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.056148 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.056645 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.056737 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.056853 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:38.057028 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 14:52:38.057107 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":90,"threads_started":1}
I20260926 14:52:38.057230 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.051291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:43161 (local address 127.0.0.1:49178)
0926 14:52:38.053077 (+ 1786us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:38.053119 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:38.053242 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:38.053707 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:38.053712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:38.053737 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:38.053859 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.053866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.054951 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.054954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.056045 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.056055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.056750 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.056753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.056856 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.056891 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 14:52:38.056988 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 14:52:38.057102 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1629,"thread_start_us":94,"threads_started":1}
I20260926 14:52:38.058048 26994 tablet_service.cc:1514] Processing DeleteTablet for tablet 493bfe58692b4866b3bf6e7296516780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:52:37 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:49178
I20260926 14:52:38.058718 27069 tablet_replica.cc:330] stopping tablet replica
I20260926 14:52:38.058959 27069 raft_consensus.cc:2229] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:52:38.059155 27069 raft_consensus.cc:2258] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:52:38.059969 27069 ts_tablet_manager.cc:1905] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:52:38.063057 27069 ts_tablet_manager.cc:1918] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:52:38.063127 27069 log.cc:1196] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal/wals/493bfe58692b4866b3bf6e7296516780
I20260926 14:52:38.063340 27069 ts_tablet_manager.cc:1939] T 493bfe58692b4866b3bf6e7296516780 P b1490811f65a45b4a3ba202047b1bd1e: Deleting consensus metadata
I20260926 14:52:38.064879 26858 catalog_manager.cc:4841] TS b1490811f65a45b4a3ba202047b1bd1e (127.12.68.193:43161): tablet 493bfe58692b4866b3bf6e7296516780 (table test-table [id=871743935fe04cd7bf35f86da5017cf9]) successfully deleted
W20260926 14:52:38.142364 27064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:38.142586 27064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:38.142624 27064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:38.146494 27064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:38.146540 27064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:38.146593 27064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:52:38.151727 27064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:36789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=39003
--tserver_master_addrs=127.12.68.254:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:38.152750 27064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:38.153684 27064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:38.162505 27077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:38.162518 27074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:38.162520 27075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:38.162675 27064 server_base.cc:1034] running on GCE node
I20260926 14:52:38.163013 27064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:38.163569 27064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:38.164714 27064 hybrid_clock.cc:648] HybridClock initialized: now 1790434358164699 us; error 26 us; skew 500 ppm
Sep 26 14:52:38 dist-test-slave-24mn krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434358, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:38.167097 27064 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:52:38.167816 27064 webserver.cc:460] Webserver started at http://127.12.68.194:39003/ using document root <none> and password file <none>
I20260926 14:52:38.168321 27064 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:38.168388 27064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:38.171820 27064 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:52:38.173671 27084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:38.174773 27064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:52:38.174882 27064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "4c6e58cd584d4fe6ab89bf89b508fe43"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "4323cb332f36b2c1f5cdb6bb152eeca3"
server_key_iv: "56715596659495e25bb391a51520a7d8"
server_key_version: "encryptionkey@0"
I20260926 14:52:38.175366 27064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:38.196867 27064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:38.199441 27064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:38.199605 27064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:38.200165 27064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:38.200915 27064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:38.200958 27064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:38.201009 27064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:38.201036 27064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:38.208766 27064 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:36789
I20260926 14:52:38.208796 27197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:36789 every 8 connection(s)
I20260926 14:52:38.209647 27064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:52:38.217355 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 27064
I20260926 14:52:38.217716 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:44993
--local_ip_for_outbound_sockets=127.12.68.195
--tserver_master_addrs=127.12.68.254:40155
--webserver_port=35075
--webserver_interface=127.12.68.195
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:52:38 dist-test-slave-24mn krb5kdc[26290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434358, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:38.221963 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.211244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:55091 (local address 127.12.68.254:40155)
0926 14:52:38.211357 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:38.211360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:38.212166 (+ 806us) server_negotiation.cc:407] Connection header received
0926 14:52:38.213432 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:38.213435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:38.213498 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:38.213565 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:38.214633 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.215138 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.216230 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.216461 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.216554 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.218660 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:38.218664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:38.218666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:38.218691 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:38.220090 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:38.220528 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:38.220531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:38.220533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:38.220607 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:38.220886 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:38.220889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:38.220891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:38.221064 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:38.221155 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:38.221747 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 14:52:38.221847 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:52:38.223033 27200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.211604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:55091)
0926 14:52:38.212042 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:38.212081 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:38.213248 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:38.213736 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:38.213742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:38.214071 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:38.214502 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.214516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.215245 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.215249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.216128 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.216136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.216593 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.216597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.216740 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.217189 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:38.217204 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:38.218516 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:38.220191 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:38.220196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:38.220209 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:38.220436 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:38.220681 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:38.220683 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:38.220685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:38.220804 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:38.221194 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:38.221201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:38.221510 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 14:52:38.221726 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 14:52:38.222048 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":104,"threads_started":1}
I20260926 14:52:38.224367 27198 heartbeater.cc:347] Connected to a master server at 127.12.68.254:40155
I20260926 14:52:38.224581 27198 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:38.225023 27198 heartbeater.cc:511] Master 127.12.68.254:40155 requested a full tablet report, sending...
I20260926 14:52:38.226042 26872 ts_manager.cc:194] Registered new tserver with Master: 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789)
I20260926 14:52:38.226796 26872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:55091
W20260926 14:52:38.322599 27201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:38.322820 27201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:38.322860 27201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:38.326682 27201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:38.326725 27201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:38.326778 27201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:52:38.331664 27201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:44993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=35075
--tserver_master_addrs=127.12.68.254:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:38.332540 27201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:38.333416 27201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:38.342291 27206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:38.342322 27207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:38.342334 27209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:38.342618 27201 server_base.cc:1034] running on GCE node
I20260926 14:52:38.342952 27201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:38.343477 27201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:38.344640 27201 hybrid_clock.cc:648] HybridClock initialized: now 1790434358344623 us; error 24 us; skew 500 ppm
Sep 26 14:52:38 dist-test-slave-24mn krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434358, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:38.347048 27201 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:52:38.347759 27201 webserver.cc:460] Webserver started at http://127.12.68.195:35075/ using document root <none> and password file <none>
I20260926 14:52:38.348253 27201 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:38.348320 27201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:38.351691 27201 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:52:38.353518 27216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:38.354605 27201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 14:52:38.354700 27201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "7d918660f51146b78e50a4ce427a31e7"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "ec8e4aeba2141f85e554a8f51de55c4b"
server_key_iv: "23ff4971f75fa7ca892bf699f279bdea"
server_key_version: "encryptionkey@0"
I20260926 14:52:38.355165 27201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:38.371559 27201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:38.374281 27201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:38.374433 27201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:38.374943 27201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:38.375725 27201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:38.375769 27201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:38.375818 27201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:38.375844 27201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:38.383911 27201 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:44993
I20260926 14:52:38.383945 27329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:44993 every 8 connection(s)
I20260926 14:52:38.384814 27201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:52:38.393348 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 27201
Sep 26 14:52:38 dist-test-slave-24mn krb5kdc[26290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434358, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:38.397312 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.386373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:40017 (local address 127.12.68.254:40155)
0926 14:52:38.386497 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:38.386501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:38.387263 (+ 762us) server_negotiation.cc:407] Connection header received
0926 14:52:38.388574 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:38.388578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:38.388632 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:38.388704 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:38.389869 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.390490 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.391556 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.391784 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.391869 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.394115 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:38.394118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:38.394121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:38.394146 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:38.395510 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:38.395971 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:38.395974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:38.395975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:38.396045 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:38.396338 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:38.396342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:38.396344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:38.396510 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:38.396590 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:38.397053 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 14:52:38.397160 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:52:38.398010 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.386741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:40017)
0926 14:52:38.387134 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:38.387171 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:38.388387 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:38.388878 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:38.388885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:38.389250 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:38.389726 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.389739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.390601 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.390604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.391447 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.391455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.391938 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.391942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.392099 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.392562 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:38.392577 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:38.393979 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:38.395606 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:38.395612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:38.395625 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:38.395847 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:38.396121 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:38.396124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:38.396126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:38.396249 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:38.396601 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:38.396607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:38.396821 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 14:52:38.396975 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 14:52:38.397267 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":113,"threads_started":1}
I20260926 14:52:38.399080 27330 heartbeater.cc:347] Connected to a master server at 127.12.68.254:40155
I20260926 14:52:38.399283 27330 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:38.399694 27330 heartbeater.cc:511] Master 127.12.68.254:40155 requested a full tablet report, sending...
I20260926 14:52:38.400662 26872 ts_manager.cc:194] Registered new tserver with Master: 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993)
I20260926 14:52:38.401192 27333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.393963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.12.68.254:40155)
0926 14:52:38.394557 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:38.394561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:38.394573 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:52:38.394606 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:38.394608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:38.394646 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:38.394717 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:38.395387 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.395927 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.396724 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.396954 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.397037 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.398343 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:38.398346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:38.398347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:38.398368 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:38.399654 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:38.400087 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:38.400091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:38.400093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:38.400164 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:38.400405 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:38.400408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:38.400410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:38.400599 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:38.400797 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:38.400937 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 14:52:38.401044 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":64,"threads_started":1}
I20260926 14:52:38.401333 26872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:40017
I20260926 14:52:38.402798 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:52:38.406495 27333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.403331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.12.68.254:40155)
0926 14:52:38.403426 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:38.403430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:38.403579 (+ 149us) server_negotiation.cc:407] Connection header received
0926 14:52:38.403724 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:38.403727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:38.403807 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:38.403910 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:52:38.404096 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.404557 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.405318 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.405538 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.405617 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.405881 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:52:38.406186 (+ 305us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:52:38.406280 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:38.406343 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 14:52:38.406393 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 14:52:38.406981 26871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:52:38.410063 26871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:52:38.416999 26994 tablet_service.cc:1467] Processing CreateTablet for tablet 94e75e7bb5fa4c6dbda772fbe71fba11 (DEFAULT_TABLE table=test-table [id=00290a9f716744828d763cec481de1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:52:38.417496 26994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e75e7bb5fa4c6dbda772fbe71fba11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:38.419821 26952 tablet_bootstrap.cc:492] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e: Bootstrap starting.
I20260926 14:52:38.420619 26952 tablet_bootstrap.cc:654] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:38.421891 26952 tablet_bootstrap.cc:492] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e: No bootstrap required, opened a new log
I20260926 14:52:38.421976 26952 ts_tablet_manager.cc:1397] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:52:38.422372 26952 raft_consensus.cc:348] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:38.422485 26952 raft_consensus.cc:374] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:38.422533 26952 raft_consensus.cc:729] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1490811f65a45b4a3ba202047b1bd1e, State: Initialized, Role: FOLLOWER
I20260926 14:52:38.422652 26952 consensus_queue.cc:260] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:38.422923 26952 ts_tablet_manager.cc:1428] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:52:38.423949 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.416048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:44993 (local address 127.0.0.1:48746)
0926 14:52:38.416157 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:38.416170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:38.416239 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:38.420393 (+ 4154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:38.420396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:38.420414 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:38.420517 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.420524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.421875 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.421878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.422720 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.422729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.423558 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.423562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.423642 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.423666 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:52:38.423718 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 14:52:38.423788 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 14:52:38.423978 27341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.416403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48746 (local address 127.12.68.195:44993)
0926 14:52:38.420033 (+ 3630us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:38.420044 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:38.420062 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:52:38.420116 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:38.420120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:38.420269 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:38.420390 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:38.420648 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.421759 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.422849 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.423443 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.423549 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.423688 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:38.423740 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 14:52:38.423822 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3542,"thread_start_us":124,"threads_started":1}
I20260926 14:52:38.425347 27339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.416269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:36789 (local address 127.0.0.1:41888)
0926 14:52:38.416519 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:38.416532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:38.416599 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:38.421260 (+ 4661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:38.421263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:38.421278 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:38.421339 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.421343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.422596 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.422599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.423602 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.423610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.425035 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.425038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.425134 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.425152 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:52:38.425206 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 14:52:38.425256 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260926 14:52:38.425596 27340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.416421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.12.68.194:36789)
0926 14:52:38.420642 (+ 4221us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:38.420653 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:38.420675 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:52:38.420748 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:38.420753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:38.420954 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:38.421099 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:38.421438 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.422445 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.423957 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.424703 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.424794 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.424921 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:38.425246 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 14:52:38.425333 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4119,"thread_start_us":106,"threads_started":1}
I20260926 14:52:38.426692 27264 tablet_service.cc:1467] Processing CreateTablet for tablet 94e75e7bb5fa4c6dbda772fbe71fba11 (DEFAULT_TABLE table=test-table [id=00290a9f716744828d763cec481de1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:52:38.427575 27264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e75e7bb5fa4c6dbda772fbe71fba11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:38.427671 27132 tablet_service.cc:1467] Processing CreateTablet for tablet 94e75e7bb5fa4c6dbda772fbe71fba11 (DEFAULT_TABLE table=test-table [id=00290a9f716744828d763cec481de1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:52:38.428561 27132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e75e7bb5fa4c6dbda772fbe71fba11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:38.433740 27343 tablet_bootstrap.cc:492] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7: Bootstrap starting.
I20260926 14:52:38.434504 27344 tablet_bootstrap.cc:492] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43: Bootstrap starting.
I20260926 14:52:38.436362 27343 tablet_bootstrap.cc:654] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:38.436954 27344 tablet_bootstrap.cc:654] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:38.437207 27343 log.cc:824] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:38.437717 27344 log.cc:824] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:38.439190 27343 tablet_bootstrap.cc:492] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7: No bootstrap required, opened a new log
I20260926 14:52:38.439388 27343 ts_tablet_manager.cc:1397] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 14:52:38.439481 27344 tablet_bootstrap.cc:492] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43: No bootstrap required, opened a new log
I20260926 14:52:38.439643 27344 ts_tablet_manager.cc:1397] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:52:38.443127 27344 raft_consensus.cc:348] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:38.443127 27343 raft_consensus.cc:348] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:38.443377 27343 raft_consensus.cc:374] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:38.443377 27344 raft_consensus.cc:374] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:38.443432 27344 raft_consensus.cc:729] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6e58cd584d4fe6ab89bf89b508fe43, State: Initialized, Role: FOLLOWER
I20260926 14:52:38.443439 27343 raft_consensus.cc:729] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d918660f51146b78e50a4ce427a31e7, State: Initialized, Role: FOLLOWER
I20260926 14:52:38.443871 27343 consensus_queue.cc:260] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:38.443872 27344 consensus_queue.cc:260] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:38.444479 27330 heartbeater.cc:503] Master 127.12.68.254:40155 was elected leader, sending a full tablet report...
I20260926 14:52:38.444775 27198 heartbeater.cc:503] Master 127.12.68.254:40155 was elected leader, sending a full tablet report...
I20260926 14:52:38.444832 27343 ts_tablet_manager.cc:1428] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260926 14:52:38.444833 27344 ts_tablet_manager.cc:1428] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
W20260926 14:52:38.460440 27199 tablet.cc:2369] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:52:38.512645 27067 raft_consensus.cc:482] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:52:38.512847 27067 raft_consensus.cc:504] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:38.513901 27067 leader_election.cc:288] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993), 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789)
I20260926 14:52:38.517323 27341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.514246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:48539 (local address 127.12.68.195:44993)
0926 14:52:38.514426 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:38.514429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:38.514440 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:52:38.514488 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:38.514491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:38.514527 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:38.514585 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:38.514835 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.515319 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.516392 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.516788 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.516891 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.516941 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:38.517139 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 14:52:38.517209 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 14:52:38.517323 27063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.514075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:44993 (local address 127.12.68.193:48539)
0926 14:52:38.514185 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:38.514200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:38.514281 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:38.514619 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:38.514622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:38.514639 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:38.514736 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.514740 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.515421 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.515424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.516279 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.516287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.516893 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.516896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.517034 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.517051 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 14:52:38.517114 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:52:38.517182 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":43}
I20260926 14:52:38.517375 27340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.514230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:56131 (local address 127.12.68.194:36789)
0926 14:52:38.514347 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:38.514352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:38.514516 (+ 164us) server_negotiation.cc:407] Connection header received
0926 14:52:38.514629 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:38.514632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:38.514677 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:38.514737 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:38.514976 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.515452 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.516495 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.516877 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.516975 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.517024 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:38.517197 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 14:52:38.517261 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:52:38.517443 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.514134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:36789 (local address 127.12.68.193:56131)
0926 14:52:38.514427 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:38.514439 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:38.514535 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:38.514755 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:38.514757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:38.514772 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:38.514885 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.514890 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.515551 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.515555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.516384 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:38.516393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.516993 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.516997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:38.517095 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.517112 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 14:52:38.517173 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:52:38.517236 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":16,"thread_start_us":65,"threads_started":1}
I20260926 14:52:38.517933 27152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e75e7bb5fa4c6dbda772fbe71fba11" candidate_uuid: "b1490811f65a45b4a3ba202047b1bd1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" is_pre_election: true
I20260926 14:52:38.517935 27284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e75e7bb5fa4c6dbda772fbe71fba11" candidate_uuid: "b1490811f65a45b4a3ba202047b1bd1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d918660f51146b78e50a4ce427a31e7" is_pre_election: true
I20260926 14:52:38.518182 27152 raft_consensus.cc:2454] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1490811f65a45b4a3ba202047b1bd1e in term 0.
I20260926 14:52:38.518193 27284 raft_consensus.cc:2454] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1490811f65a45b4a3ba202047b1bd1e in term 0.
I20260926 14:52:38.518566 26947 leader_election.cc:302] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d918660f51146b78e50a4ce427a31e7, b1490811f65a45b4a3ba202047b1bd1e; no voters:
I20260926 14:52:38.518806 27067 raft_consensus.cc:2781] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:52:38.518878 27067 raft_consensus.cc:482] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:52:38.518908 27067 raft_consensus.cc:3037] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:38.519932 27067 raft_consensus.cc:504] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:38.520239 27067 leader_election.cc:288] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 1 election: Requested vote from peers 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993), 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789)
I20260926 14:52:38.520581 27284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e75e7bb5fa4c6dbda772fbe71fba11" candidate_uuid: "b1490811f65a45b4a3ba202047b1bd1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d918660f51146b78e50a4ce427a31e7"
I20260926 14:52:38.520596 27152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e75e7bb5fa4c6dbda772fbe71fba11" candidate_uuid: "b1490811f65a45b4a3ba202047b1bd1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43"
I20260926 14:52:38.520680 27152 raft_consensus.cc:3037] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:38.520689 27284 raft_consensus.cc:3037] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:38.521590 27152 raft_consensus.cc:2454] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1490811f65a45b4a3ba202047b1bd1e in term 1.
I20260926 14:52:38.521672 27284 raft_consensus.cc:2454] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1490811f65a45b4a3ba202047b1bd1e in term 1.
I20260926 14:52:38.521838 26946 leader_election.cc:302] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6e58cd584d4fe6ab89bf89b508fe43, b1490811f65a45b4a3ba202047b1bd1e; no voters:
I20260926 14:52:38.522010 27067 raft_consensus.cc:2781] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:52:38.522205 27067 raft_consensus.cc:686] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 LEADER]: Becoming Leader. State: Replica: b1490811f65a45b4a3ba202047b1bd1e, State: Running, Role: LEADER
I20260926 14:52:38.522425 27067 consensus_queue.cc:237] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:38.525106 26872 catalog_manager.cc:5495] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e reported cstate change: term changed from 0 to 1, leader changed from <none> to b1490811f65a45b4a3ba202047b1bd1e (127.12.68.193). New cstate: current_term: 1 leader_uuid: "b1490811f65a45b4a3ba202047b1bd1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } health_report { overall_health: HEALTHY } } }
W20260926 14:52:38.528895 27061 tablet.cc:2369] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:52:38.540732 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.537738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49190 (local address 127.12.68.193:43161)
0926 14:52:38.537842 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:38.537856 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:38.537868 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:52:38.538025 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:38.538027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:38.538073 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:38.538153 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:52:38.538381 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.538848 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.539586 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:38.539781 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:38.539844 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:38.540073 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:52:38.540438 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:52:38.540525 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:38.540570 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 14:52:38.540615 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:52:38.543871 27284 raft_consensus.cc:1261] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7 [term 1 FOLLOWER]: Refusing update from remote peer b1490811f65a45b4a3ba202047b1bd1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:52:38.543870 27152 raft_consensus.cc:1261] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 1 FOLLOWER]: Refusing update from remote peer b1490811f65a45b4a3ba202047b1bd1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:52:38.544518 27350 consensus_queue.cc:1035] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:52:38.544663 27067 consensus_queue.cc:1035] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:52:38.550592 27352 mvcc.cc:205] Tried to move back new op lower bound from 7333619132590882816 to 7333619132510072832. Current Snapshot: MvccSnapshot[applied={T|T < 7333619132590882816}]
I20260926 14:52:38.550937 27353 mvcc.cc:205] Tried to move back new op lower bound from 7333619132590882816 to 7333619132510072832. Current Snapshot: MvccSnapshot[applied={T|T < 7333619132590882816}]
I20260926 14:52:38.553510 27354 mvcc.cc:205] Tried to move back new op lower bound from 7333619132590882816 to 7333619132510072832. Current Snapshot: MvccSnapshot[applied={T|T < 7333619132590882816}]
I20260926 14:52:38.558257 26871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:52:38.558403 26871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:52:38.560559 26871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Sending DeleteTablet for 3 replicas of tablet 94e75e7bb5fa4c6dbda772fbe71fba11
I20260926 14:52:38.561089 27264 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e75e7bb5fa4c6dbda772fbe71fba11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:52:38 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:48746
I20260926 14:52:38.561164 27132 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e75e7bb5fa4c6dbda772fbe71fba11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:52:38 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:41888
I20260926 14:52:38.561224 26994 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e75e7bb5fa4c6dbda772fbe71fba11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:52:38 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:49178
I20260926 14:52:38.561460 27069 tablet_replica.cc:330] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e: stopping tablet replica
I20260926 14:52:38.561482 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 26910
I20260926 14:52:38.561509 27362 tablet_replica.cc:330] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7: stopping tablet replica
I20260926 14:52:38.561575 27069 raft_consensus.cc:2229] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:52:38.561761 27362 raft_consensus.cc:2229] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:52:38.561794 27069 raft_consensus.cc:2258] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:52:38.562088 27362 raft_consensus.cc:2258] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:52:38.563002 27363 tablet_replica.cc:330] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43: stopping tablet replica
I20260926 14:52:38.563234 27363 raft_consensus.cc:2229] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:52:38.563416 27362 ts_tablet_manager.cc:1905] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:52:38.563457 27363 raft_consensus.cc:2258] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:52:38.564527 27363 ts_tablet_manager.cc:1905] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:52:38.565887 27362 ts_tablet_manager.cc:1918] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:52:38.565994 27362 log.cc:1196] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/94e75e7bb5fa4c6dbda772fbe71fba11
I20260926 14:52:38.566284 27362 ts_tablet_manager.cc:1939] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 7d918660f51146b78e50a4ce427a31e7: Deleting consensus metadata
I20260926 14:52:38.566527 27363 ts_tablet_manager.cc:1918] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:52:38.566615 27363 log.cc:1196] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/94e75e7bb5fa4c6dbda772fbe71fba11
I20260926 14:52:38.566879 27363 ts_tablet_manager.cc:1939] T 94e75e7bb5fa4c6dbda772fbe71fba11 P 4c6e58cd584d4fe6ab89bf89b508fe43: Deleting consensus metadata
I20260926 14:52:38.567337 26857 catalog_manager.cc:4841] TS 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993): tablet 94e75e7bb5fa4c6dbda772fbe71fba11 (table test-table [id=00290a9f716744828d763cec481de1e5]) successfully deleted
I20260926 14:52:38.567783 26856 catalog_manager.cc:4841] TS 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789): tablet 94e75e7bb5fa4c6dbda772fbe71fba11 (table test-table [id=00290a9f716744828d763cec481de1e5]) successfully deleted
W20260926 14:52:38.568225 26858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:43161 (error 108)
I20260926 14:52:38.568929 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 27064
I20260926 14:52:38.569085 27339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:38.568820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:43161 (local address 127.0.0.1:49192)
0926 14:52:38.568940 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:38.569011 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:43161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 14:52:38.569533 26858 catalog_manager.cc:4584] TS b1490811f65a45b4a3ba202047b1bd1e (127.12.68.193:43161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94e75e7bb5fa4c6dbda772fbe71fba11: Network error: Client connection negotiation failed: client connection to 127.12.68.193:43161: connect: Connection refused (error 111)
I20260926 14:52:38.574496 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 27201
I20260926 14:52:38.578840 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 26839
I20260926 14:52:38.583720 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:40155
--webserver_interface=127.12.68.254
--webserver_port=43661
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:38.686295 27365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:38.686525 27365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:38.686563 27365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:38.690517 27365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:52:38.690562 27365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:38.690579 27365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:38.690596 27365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:52:38.690624 27365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:52:38.695724 27365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:40155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=43661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:38.696615 27365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:38.697512 27365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:38.705632 27373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:38.705698 27370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:38.705634 27371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:38.705689 27365 server_base.cc:1034] running on GCE node
I20260926 14:52:38.706321 27365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:38.707310 27365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:38.708472 27365 hybrid_clock.cc:648] HybridClock initialized: now 1790434358708450 us; error 32 us; skew 500 ppm
Sep 26 14:52:38 dist-test-slave-24mn krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434358, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:38.710960 27365 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:52:38.711839 27365 webserver.cc:460] Webserver started at http://127.12.68.254:43661/ using document root <none> and password file <none>
I20260926 14:52:38.712339 27365 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:38.712406 27365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:38.715835 27365 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:52:38.717759 27380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:38.718849 27365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:52:38.718966 27365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "220004e75e75442ba048574197d8c6df"
format_stamp: "Formatted at 2026-09-26 14:52:36 on dist-test-slave-24mn"
server_key: "0c2c818eb1d2de7ea1b2487f3653d307"
server_key_iv: "6e63b7e282652756ac59fdb129f77721"
server_key_version: "encryptionkey@0"
I20260926 14:52:38.719473 27365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:38.748175 27365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:38.751056 27365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:38.751217 27365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:38.757692 27365 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:40155
I20260926 14:52:38.757931 27432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:40155 every 8 connection(s)
I20260926 14:52:38.758671 27365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:52:38.758876 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 27365
I20260926 14:52:38.759300 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:43161
--local_ip_for_outbound_sockets=127.12.68.193
--tserver_master_addrs=127.12.68.254:40155
--webserver_port=34983
--webserver_interface=127.12.68.193
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:52:38.763064 27433 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:52:38.765319 27433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Bootstrap starting.
I20260926 14:52:38.787127 27433 log.cc:824] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:38.800376 27433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:52:38.800758 27433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df: Bootstrap complete.
I20260926 14:52:38.805878 27433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } }
I20260926 14:52:38.806358 27433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220004e75e75442ba048574197d8c6df, State: Initialized, Role: FOLLOWER
I20260926 14:52:38.806891 27433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } }
I20260926 14:52:38.807041 27433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:52:38.807114 27433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:52:38.807206 27433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 2 FOLLOWER]: Advancing to term 3
I20260926 14:52:38.809495 27433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } }
I20260926 14:52:38.809857 27433 leader_election.cc:302] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 220004e75e75442ba048574197d8c6df; no voters:
I20260926 14:52:38.810210 27433 leader_election.cc:288] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 14:52:38.810312 27437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 3 FOLLOWER]: Leader election won for term 3
I20260926 14:52:38.810546 27437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [term 3 LEADER]: Becoming Leader. State: Replica: 220004e75e75442ba048574197d8c6df, State: Running, Role: LEADER
I20260926 14:52:38.810827 27437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } }
I20260926 14:52:38.811393 27433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:52:38.811627 27439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 220004e75e75442ba048574197d8c6df. Latest consensus state: current_term: 3 leader_uuid: "220004e75e75442ba048574197d8c6df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } } }
I20260926 14:52:38.811743 27439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:38.812029 27438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "220004e75e75442ba048574197d8c6df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220004e75e75442ba048574197d8c6df" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 40155 } } }
I20260926 14:52:38.812117 27438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 220004e75e75442ba048574197d8c6df [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:38.813284 27441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:52:38.817340 27441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:52:38.817852 27441 catalog_manager.cc:1259] Loaded cluster ID: 51e68dea7220441c96ec8a699c820187
I20260926 14:52:38.817899 27441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 14:52:38.818651 27441 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4d6bcb79ad google::LogMessage::Fail() at ??:0
@ 0x7f4d6bcba88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4d6bcb74de google::LogMessage::Flush() at ??:0
@ 0x7f4d6bcb8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4d765f6fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4d765f7d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4d765f4579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4d76638640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4d6df093d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4d6df09c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4d6df0b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4d6defa512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4d6f53e6db start_thread at ??:0
@ 0x7f4d6a7b561f clone at ??:0
W20260926 14:52:38.896096 27435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:38.896315 27435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:38.896371 27435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:38.900239 27435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:38.900285 27435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:38.900359 27435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:52:38.905550 27435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:43161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=34983
--tserver_master_addrs=127.12.68.254:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:38.906430 27435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:38.907322 27435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:38.917367 27475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:38.917426 27435 server_base.cc:1034] running on GCE node
W20260926 14:52:38.917609 27477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:38.917366 27474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:38.918045 27435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:38.918675 27435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:38.919842 27435 hybrid_clock.cc:648] HybridClock initialized: now 1790434358919817 us; error 39 us; skew 500 ppm
Sep 26 14:52:38 dist-test-slave-24mn krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434358, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:38.922425 27435 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:52:38.923250 27435 webserver.cc:460] Webserver started at http://127.12.68.193:34983/ using document root <none> and password file <none>
I20260926 14:52:38.923766 27435 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:38.923830 27435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:38.927583 27435 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:52:38.929637 27484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:38.930863 27435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:52:38.930963 27435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "b1490811f65a45b4a3ba202047b1bd1e"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "c9001a600b3ce1fde426681d5baf6d40"
server_key_iv: "a1b0b743b362324148ae0a82fae0f14b"
server_key_version: "encryptionkey@0"
I20260926 14:52:38.931456 27435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:38.948801 27435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:38.951697 27435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:38.951853 27435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:38.952386 27435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:38.953385 27491 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:52:38.958436 27435 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:52:38.958491 27435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 14:52:38.958567 27435 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:52:38.960094 27435 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:52:38.960136 27435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260926 14:52:38.960438 27491 tablet_bootstrap.cc:492] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e: Bootstrap starting.
I20260926 14:52:38.969244 27435 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:43161
I20260926 14:52:38.969502 27598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:43161 every 8 connection(s)
I20260926 14:52:38.970184 27435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:52:38.976070 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 27435
I20260926 14:52:38.976438 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:36789
--local_ip_for_outbound_sockets=127.12.68.194
--tserver_master_addrs=127.12.68.254:40155
--webserver_port=39003
--webserver_interface=127.12.68.194
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:52:38.982594 27491 log.cc:824] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:38.990952 27491 tablet_bootstrap.cc:492] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:52:38.991309 27491 tablet_bootstrap.cc:492] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e: Bootstrap complete.
I20260926 14:52:38.992599 27491 ts_tablet_manager.cc:1397] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.019s
I20260926 14:52:38.995678 27491 raft_consensus.cc:348] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:38.996238 27491 raft_consensus.cc:729] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1490811f65a45b4a3ba202047b1bd1e, State: Initialized, Role: FOLLOWER
I20260926 14:52:38.996783 27491 consensus_queue.cc:260] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:38.997777 27491 ts_tablet_manager.cc:1428] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
W20260926 14:52:39.041325 27601 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:52:38.971989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:57241)
0926 14:52:38.972421 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:38.972470 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:38.973244 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:39.040531 (+ 67287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":112,"threads_started":1}
W20260926 14:52:39.041783 27599 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:40155 (0 consecutive failures): Network error: Failed to ping master at 127.12.68.254:40155: Client connection negotiation failed: client connection to 127.12.68.254:40155: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:52:39.042512 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:39.042261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:40955)
0926 14:52:39.042394 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:39.042458 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 14:52:39.043066 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:39.042895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:55323)
0926 14:52:39.042981 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:39.043020 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 14:52:39.043215 27599 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:52:39.084164 27602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:39.084376 27602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:39.084415 27602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:39.088250 27602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:39.088296 27602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:39.088349 27602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:52:39.093281 27602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:36789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=39003
--tserver_master_addrs=127.12.68.254:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:39.094118 27602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:39.094971 27602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:39.103473 27612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:39.103482 27610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:39.103482 27609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:39.103653 27602 server_base.cc:1034] running on GCE node
I20260926 14:52:39.104135 27602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:39.104678 27602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:39.105821 27602 hybrid_clock.cc:648] HybridClock initialized: now 1790434359105797 us; error 36 us; skew 500 ppm
Sep 26 14:52:39 dist-test-slave-24mn krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434359, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:39.108251 27602 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:52:39.109001 27602 webserver.cc:460] Webserver started at http://127.12.68.194:39003/ using document root <none> and password file <none>
I20260926 14:52:39.109459 27602 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:39.109512 27602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:39.112897 27602 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 14:52:39.114739 27619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:39.115792 27602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260926 14:52:39.115913 27602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "4c6e58cd584d4fe6ab89bf89b508fe43"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "4323cb332f36b2c1f5cdb6bb152eeca3"
server_key_iv: "56715596659495e25bb391a51520a7d8"
server_key_version: "encryptionkey@0"
I20260926 14:52:39.116415 27602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:39.149307 27602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:39.152256 27602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:39.152421 27602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:39.152968 27602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:39.153754 27602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:39.153800 27602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:39.153851 27602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:39.153878 27602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:39.162179 27602 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:36789
I20260926 14:52:39.162209 27732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:36789 every 8 connection(s)
I20260926 14:52:39.163101 27602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:52:39.163933 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 27602
I20260926 14:52:39.164355 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:44993
--local_ip_for_outbound_sockets=127.12.68.195
--tserver_master_addrs=127.12.68.254:40155
--webserver_port=35075
--webserver_interface=127.12.68.195
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:52:39.166865 27736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:39.165221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:37403)
0926 14:52:39.165686 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:39.165787 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":131,"threads_started":1}
W20260926 14:52:39.167266 27733 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:40155 (0 consecutive failures): Network error: Failed to ping master at 127.12.68.254:40155: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
I20260926 14:52:39.167807 27736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:39.167613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:45321)
0926 14:52:39.167712 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:39.167756 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 14:52:39.168376 27736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:39.168200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:60741)
0926 14:52:39.168290 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:39.168330 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 14:52:39.168511 27733 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:52:39.267120 27735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:39.267338 27735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:39.267391 27735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:39.271318 27735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:39.271368 27735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:39.271445 27735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:52:39.276335 27735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:44993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=35075
--tserver_master_addrs=127.12.68.254:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:39.277194 27735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:39.278079 27735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:39.286674 27744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:39.286710 27735 server_base.cc:1034] running on GCE node
W20260926 14:52:39.286674 27741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:39.286705 27742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:39.287242 27735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:39.287783 27735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:39.288928 27735 hybrid_clock.cc:648] HybridClock initialized: now 1790434359288915 us; error 26 us; skew 500 ppm
Sep 26 14:52:39 dist-test-slave-24mn krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434359, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:39.291352 27735 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:52:39.292136 27735 webserver.cc:460] Webserver started at http://127.12.68.195:35075/ using document root <none> and password file <none>
I20260926 14:52:39.292618 27735 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:39.292680 27735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:39.296062 27735 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:52:39.297907 27751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:39.299017 27735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:52:39.299124 27735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "7d918660f51146b78e50a4ce427a31e7"
format_stamp: "Formatted at 2026-09-26 14:52:37 on dist-test-slave-24mn"
server_key: "ec8e4aeba2141f85e554a8f51de55c4b"
server_key_iv: "23ff4971f75fa7ca892bf699f279bdea"
server_key_version: "encryptionkey@0"
I20260926 14:52:39.299600 27735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:39.326195 27735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:39.328677 27735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:39.328828 27735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:39.329334 27735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:39.330070 27735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:39.330114 27735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:39.330163 27735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:39.330189 27735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:39.337939 27735 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:44993
I20260926 14:52:39.337991 27864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:44993 every 8 connection(s)
I20260926 14:52:39.338804 27735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:52:39.340219 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 27735
I20260926 14:52:39.342289 27868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:39.340649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:50657)
0926 14:52:39.341080 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:39.341203 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
W20260926 14:52:39.342658 27865 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:40155 (0 consecutive failures): Network error: Failed to ping master at 127.12.68.254:40155: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
I20260926 14:52:39.343118 27868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:39.342935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:55081)
0926 14:52:39.343027 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:39.343068 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 14:52:39.343614 27868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:39.343457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:60087)
0926 14:52:39.343532 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:39.343572 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 14:52:39.343757 27865 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:52:40.044116 27869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:40.043701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:53495)
0926 14:52:40.043959 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:40.044010 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20260926 14:52:40.169391 27870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:40.169037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:49913)
0926 14:52:40.169248 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:40.169301 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":67,"threads_started":1}
I20260926 14:52:40.344573 27871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:40.344203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:44161)
0926 14:52:40.344425 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:40.344481 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":58,"threads_started":1}
I20260926 14:52:40.346297 27872 raft_consensus.cc:482] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:52:40.346463 27872 raft_consensus.cc:504] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:40.347419 27872 leader_election.cc:288] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993), 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789)
Sep 26 14:52:40 dist-test-slave-24mn krb5kdc[26290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434358, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.194@KRBTEST.COM
Sep 26 14:52:40 dist-test-slave-24mn krb5kdc[26290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434358, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.195@KRBTEST.COM
I20260926 14:52:40.362538 27873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:40.347740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:52081 (local address 127.12.68.195:44993)
0926 14:52:40.348077 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:40.348101 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:40.348142 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:52:40.348967 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:40.348975 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:40.349429 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:40.349746 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:40.351334 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:40.352892 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:40.354594 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:40.355288 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:40.355442 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:40.358638 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:40.358643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:40.358656 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:40.358687 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:40.360301 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:40.360803 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:40.360808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:40.360813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:40.360917 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:40.361204 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:40.361207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:40.361209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:40.361568 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:40.361678 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:40.362158 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 14:52:40.362386 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":69,"threads_started":1}
I20260926 14:52:40.362529 27869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:40.347627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:44993 (local address 127.12.68.193:52081)
0926 14:52:40.347751 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:40.347765 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:40.347851 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:40.349831 (+ 1980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:40.349837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:40.350400 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:40.351171 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:40.351191 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:40.353000 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:40.353004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:40.354462 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:40.354472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:40.355414 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:40.355418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:40.355703 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:40.356363 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:40.356375 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:40.358462 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:40.360420 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:40.360424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:40.360426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:40.360705 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:40.360999 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:40.361002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:40.361003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:40.361123 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:40.361670 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:40.361675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:40.361890 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 14:52:40.362036 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 14:52:40.362340 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":258,"spinlock_wait_cycles":19456}
I20260926 14:52:40.362572 27875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:40.347643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:36789 (local address 127.12.68.193:43807)
0926 14:52:40.348240 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:40.348251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:40.348333 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:40.349831 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:40.349837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:40.350403 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:40.351171 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:40.351190 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:40.353000 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:40.353004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:40.354462 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:40.354472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:40.355414 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:40.355418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:40.355702 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:40.356337 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:40.356369 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:40.358167 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:40.360332 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:40.360338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:40.360352 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:40.360717 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:40.361033 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:40.361036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:40.361038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:40.361127 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:40.361672 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:40.361678 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:40.361891 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 14:52:40.362037 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 14:52:40.362413 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":739,"spinlock_wait_cycles":9600,"thread_start_us":62,"threads_started":1}
I20260926 14:52:40.362571 27874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:40.347812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:43807 (local address 127.12.68.194:36789)
0926 14:52:40.348089 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:40.348106 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:40.348328 (+ 222us) server_negotiation.cc:407] Connection header received
0926 14:52:40.349130 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:40.349137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:40.349428 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:40.349746 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:40.351334 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:40.352868 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:40.354598 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:40.355288 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:40.355442 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:40.358345 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:40.358351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:40.358362 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:40.358388 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:40.360177 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:40.360824 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:40.360829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:40.360833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:40.360942 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:40.361210 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:40.361213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:40.361214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:40.361569 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:40.361679 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:40.362155 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 14:52:40.362384 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
W20260926 14:52:40.364837 27486 leader_election.cc:341] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993): Not found: Tablet not found: 94e75e7bb5fa4c6dbda772fbe71fba11
W20260926 14:52:40.365083 27485 leader_election.cc:341] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789): Not found: Tablet not found: 94e75e7bb5fa4c6dbda772fbe71fba11
I20260926 14:52:40.365161 27485 leader_election.cc:302] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b1490811f65a45b4a3ba202047b1bd1e; no voters: 4c6e58cd584d4fe6ab89bf89b508fe43, 7d918660f51146b78e50a4ce427a31e7
I20260926 14:52:40.365391 27872 raft_consensus.cc:2726] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:52:41.045104 27876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:41.044733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:50301)
0926 14:52:41.044949 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:41.045005 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260926 14:52:41.170332 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:41.169982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:55725)
0926 14:52:41.170201 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:41.170250 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
I20260926 14:52:41.345501 27878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:41.345168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:59463)
0926 14:52:41.345377 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:41.345425 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":67,"threads_started":1}
I20260926 14:52:41.989604 27879 raft_consensus.cc:482] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:52:41.989717 27879 raft_consensus.cc:504] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:41.990090 27879 leader_election.cc:288] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993), 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789)
W20260926 14:52:41.990804 27486 leader_election.cc:341] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993): Not found: Tablet not found: 94e75e7bb5fa4c6dbda772fbe71fba11
W20260926 14:52:41.990917 27485 leader_election.cc:341] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789): Not found: Tablet not found: 94e75e7bb5fa4c6dbda772fbe71fba11
I20260926 14:52:41.990991 27485 leader_election.cc:302] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b1490811f65a45b4a3ba202047b1bd1e; no voters: 4c6e58cd584d4fe6ab89bf89b508fe43, 7d918660f51146b78e50a4ce427a31e7
I20260926 14:52:41.991119 27879 raft_consensus.cc:2726] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:52:42.046111 27880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:42.045744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:55131)
0926 14:52:42.045964 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:42.046016 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":68,"threads_started":1}
I20260926 14:52:42.171397 27881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:42.170994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:42115)
0926 14:52:42.171248 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:42.171301 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260926 14:52:42.346455 27882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:42.346089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:42489)
0926 14:52:42.346308 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:42.346359 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260926 14:52:43.047134 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:43.046754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:51387)
0926 14:52:43.046986 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:43.047040 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
I20260926 14:52:43.172421 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:43.172046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:40031)
0926 14:52:43.172281 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:43.172333 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
I20260926 14:52:43.347434 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:43.347090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:53753)
0926 14:52:43.347296 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:43.347350 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260926 14:52:44.048352 27886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:44.047954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:53509)
0926 14:52:44.048208 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:44.048257 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":64,"threads_started":1}
I20260926 14:52:44.173576 27887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:44.173145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:53953)
0926 14:52:44.173402 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:44.173476 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
I20260926 14:52:44.348436 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:44.348070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:57963)
0926 14:52:44.348299 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:44.348348 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":81,"threads_started":1}
I20260926 14:52:44.944705 27889 raft_consensus.cc:482] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:52:44.944833 27889 raft_consensus.cc:504] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:44.945245 27889 leader_election.cc:288] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993), 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789)
W20260926 14:52:44.945998 27485 leader_election.cc:341] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789): Not found: Tablet not found: 94e75e7bb5fa4c6dbda772fbe71fba11
W20260926 14:52:44.946141 27486 leader_election.cc:341] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993): Not found: Tablet not found: 94e75e7bb5fa4c6dbda772fbe71fba11
I20260926 14:52:44.946235 27486 leader_election.cc:302] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b1490811f65a45b4a3ba202047b1bd1e; no voters: 4c6e58cd584d4fe6ab89bf89b508fe43, 7d918660f51146b78e50a4ce427a31e7
I20260926 14:52:44.946393 27889 raft_consensus.cc:2726] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:52:45.049301 27890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:45.048940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:45169)
0926 14:52:45.049160 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:45.049210 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
I20260926 14:52:45.174587 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:45.174177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:50941)
0926 14:52:45.174441 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:45.174499 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":68,"threads_started":1}
I20260926 14:52:45.349396 27892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:45.349053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:34245)
0926 14:52:45.349266 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:45.349314 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260926 14:52:46.050326 27893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:46.049898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:41811)
0926 14:52:46.050178 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:46.050232 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
I20260926 14:52:46.175560 27894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:46.175181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:58573)
0926 14:52:46.175425 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:46.175475 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":64,"threads_started":1}
I20260926 14:52:46.350541 27895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:46.350075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:58811)
0926 14:52:46.350355 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:46.350406 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260926 14:52:47.051460 27896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:47.051017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:40553)
0926 14:52:47.051314 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:47.051366 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":86,"threads_started":1}
I20260926 14:52:47.176740 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:47.176287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:35023)
0926 14:52:47.176582 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:47.176641 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
I20260926 14:52:47.351590 27898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:47.351221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:55405)
0926 14:52:47.351448 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:47.351502 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
I20260926 14:52:48.052609 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:48.052198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:42597)
0926 14:52:48.052461 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:48.052517 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":82,"threads_started":1}
I20260926 14:52:48.178002 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:48.177528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:42759)
0926 14:52:48.177816 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:48.177874 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":74,"threads_started":1}
I20260926 14:52:48.352768 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:48.352318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:36429)
0926 14:52:48.352623 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:48.352678 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":85,"threads_started":1}
I20260926 14:52:48.442811 27902 raft_consensus.cc:482] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:52:48.442934 27902 raft_consensus.cc:504] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d918660f51146b78e50a4ce427a31e7" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44993 } } peers { permanent_uuid: "4c6e58cd584d4fe6ab89bf89b508fe43" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36789 } } peers { permanent_uuid: "b1490811f65a45b4a3ba202047b1bd1e" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 43161 } }
I20260926 14:52:48.443327 27902 leader_election.cc:288] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993), 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789)
W20260926 14:52:48.444069 27485 leader_election.cc:341] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c6e58cd584d4fe6ab89bf89b508fe43 (127.12.68.194:36789): Not found: Tablet not found: 94e75e7bb5fa4c6dbda772fbe71fba11
W20260926 14:52:48.444191 27486 leader_election.cc:341] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7d918660f51146b78e50a4ce427a31e7 (127.12.68.195:44993): Not found: Tablet not found: 94e75e7bb5fa4c6dbda772fbe71fba11
I20260926 14:52:48.444242 27486 leader_election.cc:302] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b1490811f65a45b4a3ba202047b1bd1e; no voters: 4c6e58cd584d4fe6ab89bf89b508fe43, 7d918660f51146b78e50a4ce427a31e7
I20260926 14:52:48.444365 27902 raft_consensus.cc:2726] T 94e75e7bb5fa4c6dbda772fbe71fba11 P b1490811f65a45b4a3ba202047b1bd1e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:52:49.053830 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:49.053410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:55455)
0926 14:52:49.053678 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:49.053734 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
I20260926 14:52:49.179159 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:49.178741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:56349)
0926 14:52:49.179003 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:49.179065 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
I20260926 14:52:49.353839 27905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:49.353478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:47847)
0926 14:52:49.353696 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:49.353748 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":65,"threads_started":1}
I20260926 14:52:50.054946 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:50.054549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:38509)
0926 14:52:50.054795 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:50.054850 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260926 14:52:50.180313 27907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:50.179855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:36019)
0926 14:52:50.180157 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:50.180220 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":68,"threads_started":1}
I20260926 14:52:50.354872 27908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:50.354472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:36297)
0926 14:52:50.354721 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:50.354781 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
I20260926 14:52:51.056262 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:51.055806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:43639)
0926 14:52:51.056095 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:51.056162 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":115,"threads_started":1}
I20260926 14:52:51.181511 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:51.181080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:38499)
0926 14:52:51.181362 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:51.181416 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
I20260926 14:52:51.356127 27911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:51.355643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:49241)
0926 14:52:51.355951 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:51.356028 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
I20260926 14:52:52.057363 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:52.056962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:40449)
0926 14:52:52.057207 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:52.057263 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
I20260926 14:52:52.182554 27913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:52.182178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:50453)
0926 14:52:52.182404 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:52.182456 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260926 14:52:52.357276 27914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:52.356853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:60033)
0926 14:52:52.357124 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:52.357184 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
I20260926 14:52:53.058674 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:53.058227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:43523)
0926 14:52:53.058503 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:53.058574 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260926 14:52:53.183758 27916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:53.183303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:35787)
0926 14:52:53.183570 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:53.183631 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20260926 14:52:53.358410 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:53.358003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.195:59317)
0926 14:52:53.358262 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:53.358319 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260926 14:52:54.060005 27918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:54.059577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.193:60393)
0926 14:52:54.059827 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:54.059902 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
I20260926 14:52:54.184994 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:54.184634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:40155 (local address 127.12.68.194:39661)
0926 14:52:54.184851 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:54.184902 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":63,"threads_started":1}
I20260926 14:52:54.345114 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 27435
I20260926 14:52:54.350188 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 27602
I20260926 14:52:54.354624 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 27735
2026-09-26T14:52:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18724 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:52:54 dist-test-slave-24mn krb5kdc[27922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:52:54 dist-test-slave-24mn krb5kdc[27922](info): set up 2 sockets
Sep 26 14:52:54 dist-test-slave-24mn krb5kdc[27922](info): commencing operation
krb5kdc: starting...
W20260926 14:52:55.307701 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:52:55 dist-test-slave-24mn krb5kdc[27922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:52:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:52:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:52:55.416991 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:45237
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:45237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:55.518065 27935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:55.518299 27935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:55.518338 27935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:55.522291 27935 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 14:52:55.522336 27935 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 14:52:55.522349 27935 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 14:52:55.522362 27935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:52:55.522392 27935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:55.522423 27935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:55.522449 27935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:52:55.522476 27935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:52:55.527508 27935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:45237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:45237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:55.528405 27935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:55.529299 27935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:55.536980 27943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:55.537009 27940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:55.537114 27941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:55.537209 27935 server_base.cc:1034] running on GCE node
I20260926 14:52:55.537789 27935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:55.538775 27935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:55.539948 27935 hybrid_clock.cc:648] HybridClock initialized: now 1790434375539934 us; error 31 us; skew 500 ppm
Sep 26 14:52:55 dist-test-slave-24mn krb5kdc[27922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434375, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:55.542430 27935 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:52:55.543222 27935 webserver.cc:460] Webserver started at http://127.12.68.254:43873/ using document root <none> and password file <none>
I20260926 14:52:55.543715 27935 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:55.543787 27935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:55.543996 27935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:52:55.545859 27935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "b999bf78bccc4b56af25a62df1afc8c9"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "62f7f6820ec4b62dac931243caf0bd2f"
server_key_iv: "9047f23f5331b547aec272c8b62ce9a3"
server_key_version: "encryptionkey@0"
I20260926 14:52:55.546351 27935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "b999bf78bccc4b56af25a62df1afc8c9"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "62f7f6820ec4b62dac931243caf0bd2f"
server_key_iv: "9047f23f5331b547aec272c8b62ce9a3"
server_key_version: "encryptionkey@0"
I20260926 14:52:55.549738 27935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:52:55.551978 27950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:55.553130 27935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:52:55.553236 27935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "b999bf78bccc4b56af25a62df1afc8c9"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "62f7f6820ec4b62dac931243caf0bd2f"
server_key_iv: "9047f23f5331b547aec272c8b62ce9a3"
server_key_version: "encryptionkey@0"
I20260926 14:52:55.553326 27935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:55.570501 27935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:55.573122 27935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:55.573282 27935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:55.579512 27935 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:45237
I20260926 14:52:55.579773 28002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:45237 every 8 connection(s)
I20260926 14:52:55.580399 27935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:52:55.581601 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 27935
I20260926 14:52:55.581709 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:52:55.581983 12563 external_mini_cluster.cc:1428] Setting key 48dddca824ee9c0786b93869e0da9705
I20260926 14:52:55.583240 28003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:55.589478 28003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Bootstrap starting.
Sep 26 14:52:55 dist-test-slave-24mn krb5kdc[27922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:55.591828 28003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:55.592617 28003 log.cc:824] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:55.594553 28003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: No bootstrap required, opened a new log
I20260926 14:52:55.595702 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:55.583228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.12.68.254:45237)
0926 14:52:55.583629 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:55.583647 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:55.583686 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:52:55.584531 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:55.584551 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:55.584930 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:55.585269 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:55.585877 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:55.587311 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:55.588165 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:55.588819 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:55.588965 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:55.591336 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:55.591341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:55.591352 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:55.591376 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:55.592994 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:55.593434 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:55.593439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:55.593444 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:55.593564 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:55.593783 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:55.593786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:55.593787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:55.594226 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:55.594424 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:55.594586 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 14:52:55.594855 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260926 14:52:55.598800 28003 raft_consensus.cc:348] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } }
I20260926 14:52:55.598986 28003 raft_consensus.cc:374] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:55.599018 28003 raft_consensus.cc:729] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b999bf78bccc4b56af25a62df1afc8c9, State: Initialized, Role: FOLLOWER
I20260926 14:52:55.599359 28003 consensus_queue.cc:260] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } }
I20260926 14:52:55.599469 28003 raft_consensus.cc:388] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:52:55.599512 28003 raft_consensus.cc:482] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:52:55.599577 28003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:55.600339 28003 raft_consensus.cc:504] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } }
I20260926 14:52:55.600601 28003 leader_election.cc:302] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b999bf78bccc4b56af25a62df1afc8c9; no voters:
I20260926 14:52:55.600826 28003 leader_election.cc:288] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:52:55.600922 28008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:52:55.601150 28008 raft_consensus.cc:686] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 1 LEADER]: Becoming Leader. State: Replica: b999bf78bccc4b56af25a62df1afc8c9, State: Running, Role: LEADER
I20260926 14:52:55.601425 28008 consensus_queue.cc:237] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } }
I20260926 14:52:55.601917 28003 sys_catalog.cc:564] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:52:55.603010 28010 sys_catalog.cc:455] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b999bf78bccc4b56af25a62df1afc8c9. Latest consensus state: current_term: 1 leader_uuid: "b999bf78bccc4b56af25a62df1afc8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } } }
I20260926 14:52:55.603111 28010 sys_catalog.cc:458] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:52:55.602988 28009 sys_catalog.cc:455] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b999bf78bccc4b56af25a62df1afc8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } } }
I20260926 14:52:55.603364 28009 sys_catalog.cc:458] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: This master's current role is: LEADER
W20260926 14:52:55.606038 28023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:52:55.606101 28023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:52:55.606220 28018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:52:55.607082 28018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:52:55.611996 28018 catalog_manager.cc:1347] Generated new cluster ID: c05b244356ea4c8daa63becf549718a3
I20260926 14:52:55.612058 28018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:52:55.638557 28018 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:52:55.639510 28018 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:52:55.645612 28018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Generated new TSK 0
I20260926 14:52:55.646206 28018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:52:55.673350 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45237
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:55.775856 28029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:55.776080 28029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:55.776126 28029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:55.779956 28029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:55.779999 28029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:55.780064 28029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:52:55.784946 28029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:55.785816 28029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:55.786691 28029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:55.795609 28034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:55.795710 28037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:55.795603 28035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:55.796314 28029 server_base.cc:1034] running on GCE node
I20260926 14:52:55.796672 28029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:55.797264 28029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:55.798431 28029 hybrid_clock.cc:648] HybridClock initialized: now 1790434375798405 us; error 39 us; skew 500 ppm
Sep 26 14:52:55 dist-test-slave-24mn krb5kdc[27922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434375, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:55.800935 28029 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:52:55.801734 28029 webserver.cc:460] Webserver started at http://127.12.68.193:42411/ using document root <none> and password file <none>
I20260926 14:52:55.802286 28029 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:55.802350 28029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:55.802532 28029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:52:55.804477 28029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "f69c45c6a4d946569810a68678df60a6"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "55cc0d15105a14b142b4bd6bd67dd96e"
server_key_iv: "d3e6c17ed933429f5917bbcc2c58ddea"
server_key_version: "encryptionkey@0"
I20260926 14:52:55.804978 28029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "f69c45c6a4d946569810a68678df60a6"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "55cc0d15105a14b142b4bd6bd67dd96e"
server_key_iv: "d3e6c17ed933429f5917bbcc2c58ddea"
server_key_version: "encryptionkey@0"
I20260926 14:52:55.808343 28029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:52:55.810534 28044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:55.811619 28029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:52:55.811721 28029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "f69c45c6a4d946569810a68678df60a6"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "55cc0d15105a14b142b4bd6bd67dd96e"
server_key_iv: "d3e6c17ed933429f5917bbcc2c58ddea"
server_key_version: "encryptionkey@0"
I20260926 14:52:55.811816 28029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:55.828052 28029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:55.830499 28029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:55.830659 28029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:55.831175 28029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:55.831966 28029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:55.832013 28029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:55.832062 28029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:55.832091 28029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:55.839968 28029 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:35281
I20260926 14:52:55.839993 28157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:35281 every 8 connection(s)
I20260926 14:52:55.840744 28029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:52:55.848424 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 28029
I20260926 14:52:55.848536 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:52:55.848798 12563 external_mini_cluster.cc:1428] Setting key 7fe6273f3a703e9b689e9741fc57f344
Sep 26 14:52:55 dist-test-slave-24mn krb5kdc[27922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434375, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:55.853379 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:55.842382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:50541 (local address 127.12.68.254:45237)
0926 14:52:55.842514 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:55.842518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:55.843280 (+ 762us) server_negotiation.cc:407] Connection header received
0926 14:52:55.844506 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:55.844510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:55.844563 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:55.844634 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:55.845747 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:55.846267 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:55.847348 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:55.847571 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:55.847658 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:55.850294 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:55.850298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:55.850301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:55.850324 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:55.851480 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:55.851963 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:55.851967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:55.851969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:55.852046 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:55.852362 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:55.852365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:55.852366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:55.852530 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:55.852612 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:55.853153 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 14:52:55.853263 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:52:55.854354 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:55.842678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:50541)
0926 14:52:55.843132 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:55.843172 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:55.844311 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:55.844828 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:55.844835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:55.845164 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:55.845594 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:55.845606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:55.846381 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:55.846384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:55.847228 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:55.847237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:55.847705 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:55.847708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:55.847857 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:55.848430 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:55.848446 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:55.850148 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:55.851587 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:55.851593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:55.851606 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:55.851854 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:55.852135 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:55.852138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:55.852140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:55.852268 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:55.852626 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:55.852632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:55.852900 (+ 268us) client_negotiation.cc:769] Sending connection context
0926 14:52:55.853080 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 14:52:55.853391 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":105,"threads_started":1}
I20260926 14:52:55.855428 28158 heartbeater.cc:347] Connected to a master server at 127.12.68.254:45237
I20260926 14:52:55.855675 28158 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:55.856263 28158 heartbeater.cc:511] Master 127.12.68.254:45237 requested a full tablet report, sending...
I20260926 14:52:55.857681 27967 ts_manager.cc:194] Registered new tserver with Master: f69c45c6a4d946569810a68678df60a6 (127.12.68.193:35281)
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
I20260926 14:52:55.858760 27967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:50541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:52:55.873863 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45237
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:55.975926 28163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:55.976135 28163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:55.976173 28163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:55.979946 28163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:55.979987 28163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:55.980038 28163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:52:55.984844 28163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:55.985649 28163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:55.986495 28163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:55.995093 28168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:55.995083 28169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:55.995085 28171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:55.995244 28163 server_base.cc:1034] running on GCE node
I20260926 14:52:55.995676 28163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:55.996258 28163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:55.997427 28163 hybrid_clock.cc:648] HybridClock initialized: now 1790434375997413 us; error 30 us; skew 500 ppm
Sep 26 14:52:55 dist-test-slave-24mn krb5kdc[27922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434375, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:55.999857 28163 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:52:56.000628 28163 webserver.cc:460] Webserver started at http://127.12.68.194:35533/ using document root <none> and password file <none>
I20260926 14:52:56.001078 28163 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:56.001129 28163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:56.001286 28163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:52:56.003168 28163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "f1bed65d37794758b8a05c6bf9196d60"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "f628e21e37a9bcf4f2966b8162850f33"
server_key_iv: "0a47472a4f95d6d57ad9d0a0ed9fc6c0"
server_key_version: "encryptionkey@0"
I20260926 14:52:56.003659 28163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "f1bed65d37794758b8a05c6bf9196d60"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "f628e21e37a9bcf4f2966b8162850f33"
server_key_iv: "0a47472a4f95d6d57ad9d0a0ed9fc6c0"
server_key_version: "encryptionkey@0"
I20260926 14:52:56.006865 28163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:52:56.008989 28178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:56.010039 28163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:52:56.010144 28163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "f1bed65d37794758b8a05c6bf9196d60"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "f628e21e37a9bcf4f2966b8162850f33"
server_key_iv: "0a47472a4f95d6d57ad9d0a0ed9fc6c0"
server_key_version: "encryptionkey@0"
I20260926 14:52:56.010231 28163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:56.034875 28163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:56.037470 28163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:56.037608 28163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:56.038092 28163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:56.038817 28163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:56.038861 28163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:56.038897 28163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:56.038916 28163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:56.046693 28163 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:42911
I20260926 14:52:56.046710 28291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:42911 every 8 connection(s)
I20260926 14:52:56.047472 28163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:52:56.048805 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 28163
I20260926 14:52:56.048923 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:52:56.049223 12563 external_mini_cluster.cc:1428] Setting key dc02c8341d8396ded8bc41ab48af2519
Sep 26 14:52:56 dist-test-slave-24mn krb5kdc[27922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434375, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
I20260926 14:52:56.060268 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.049101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:59243 (local address 127.12.68.254:45237)
0926 14:52:56.049251 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:56.049255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:56.050537 (+ 1282us) server_negotiation.cc:407] Connection header received
0926 14:52:56.051387 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:56.051390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:56.051436 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:56.051513 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:56.052688 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.053203 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.054282 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.054495 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.054576 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.056997 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:56.057001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:56.057004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:56.057032 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:56.058442 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:56.058881 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:56.058884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:56.058886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:56.058957 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:56.059256 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:56.059259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:56.059261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:56.059420 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:56.059502 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:56.059991 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 14:52:56.060103 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:52:56.060995 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.049494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:59243)
0926 14:52:56.050397 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:56.050444 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:56.051194 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:56.051669 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:56.051675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:56.052098 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:56.052560 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:56.052572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.053315 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.053318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:56.054161 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:56.054170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.054616 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.054620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:56.054773 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.055250 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:56.055265 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:56.056838 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:56.058542 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:56.058548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:56.058560 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:56.058795 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:56.059034 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:56.059037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:56.059039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:56.059161 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:56.059514 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:56.059521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:56.059745 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 14:52:56.059950 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 14:52:56.060238 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":568,"threads_started":1}
I20260926 14:52:56.062039 28292 heartbeater.cc:347] Connected to a master server at 127.12.68.254:45237
I20260926 14:52:56.062270 28292 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:56.062698 28292 heartbeater.cc:511] Master 127.12.68.254:45237 requested a full tablet report, sending...
I20260926 14:52:56.063663 27967 ts_manager.cc:194] Registered new tserver with Master: f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911)
I20260926 14:52:56.064214 27967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:59243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:52:56.072130 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45237
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:52:56.175325 28297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:52:56.175551 28297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:52:56.175596 28297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:52:56.179490 28297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:52:56.179540 28297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:52:56.179601 28297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:52:56.184609 28297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:52:56.185515 28297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:52:56.186422 28297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:52:56.195147 28302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:56.195202 28303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:52:56.195120 28305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:52:56.195233 28297 server_base.cc:1034] running on GCE node
I20260926 14:52:56.195739 28297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:52:56.196338 28297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:52:56.197499 28297 hybrid_clock.cc:648] HybridClock initialized: now 1790434376197473 us; error 38 us; skew 500 ppm
Sep 26 14:52:56 dist-test-slave-24mn krb5kdc[27922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434376, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:52:56.199950 28297 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:52:56.200695 28297 webserver.cc:460] Webserver started at http://127.12.68.195:40273/ using document root <none> and password file <none>
I20260926 14:52:56.201175 28297 fs_manager.cc:359] Metadata directory not provided
I20260926 14:52:56.201236 28297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:52:56.201400 28297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:52:56.203392 28297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "259cbbb3436140beb73e9ac00a6eb9e6"
format_stamp: "Formatted at 2026-09-26 14:52:56 on dist-test-slave-24mn"
server_key: "d6386eb578f33ac3f363874c7865ab62"
server_key_iv: "0368d0b053cf65ce5fe8d6b472db736a"
server_key_version: "encryptionkey@0"
I20260926 14:52:56.203938 28297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "259cbbb3436140beb73e9ac00a6eb9e6"
format_stamp: "Formatted at 2026-09-26 14:52:56 on dist-test-slave-24mn"
server_key: "d6386eb578f33ac3f363874c7865ab62"
server_key_iv: "0368d0b053cf65ce5fe8d6b472db736a"
server_key_version: "encryptionkey@0"
I20260926 14:52:56.207288 28297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:52:56.209452 28312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:56.210531 28297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 14:52:56.210625 28297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "259cbbb3436140beb73e9ac00a6eb9e6"
format_stamp: "Formatted at 2026-09-26 14:52:56 on dist-test-slave-24mn"
server_key: "d6386eb578f33ac3f363874c7865ab62"
server_key_iv: "0368d0b053cf65ce5fe8d6b472db736a"
server_key_version: "encryptionkey@0"
I20260926 14:52:56.210719 28297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:52:56.233315 28297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:52:56.236037 28297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:52:56.236208 28297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:52:56.236747 28297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:52:56.237506 28297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:52:56.237551 28297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:56.237599 28297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:52:56.237628 28297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:52:56.245630 28297 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:33757
I20260926 14:52:56.245671 28425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:33757 every 8 connection(s)
I20260926 14:52:56.246434 28297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:52:56.247020 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 28297
I20260926 14:52:56.247123 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:52:56.247372 12563 external_mini_cluster.cc:1428] Setting key fc12449f52d910e9d949ad66524f8148
Sep 26 14:52:56 dist-test-slave-24mn krb5kdc[27922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434376, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:52:56.259004 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.248156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:41887 (local address 127.12.68.254:45237)
0926 14:52:56.248279 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:56.248282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:56.249228 (+ 946us) server_negotiation.cc:407] Connection header received
0926 14:52:56.250511 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:56.250514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:56.250558 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:56.250628 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:56.251690 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.252254 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.253353 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.253572 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.253652 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.255809 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:56.255812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:56.255814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:56.255837 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:56.257184 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:56.257647 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:56.257651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:56.257653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:56.257737 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:56.258024 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:56.258027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:56.258029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:56.258201 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:56.258284 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:56.258733 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 14:52:56.258848 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:52:56.259792 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.248636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:41887)
0926 14:52:56.249096 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:56.249145 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:56.250315 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:56.250754 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:56.250760 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:56.251085 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:52:56.251538 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:56.251551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.252359 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.252362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:56.253216 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:56.253224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.253682 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.253684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:56.253820 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.254329 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:52:56.254344 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:52:56.255662 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:52:56.257288 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:56.257293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:56.257306 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:56.257562 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:56.257814 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:52:56.257817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:52:56.257819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:52:56.257943 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:52:56.258291 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:52:56.258297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:52:56.258516 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 14:52:56.258669 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 14:52:56.258960 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":103,"threads_started":1}
I20260926 14:52:56.260854 28426 heartbeater.cc:347] Connected to a master server at 127.12.68.254:45237
I20260926 14:52:56.261078 28426 heartbeater.cc:464] Registering TS with master...
I20260926 14:52:56.261523 28426 heartbeater.cc:511] Master 127.12.68.254:45237 requested a full tablet report, sending...
I20260926 14:52:56.262429 27967 ts_manager.cc:194] Registered new tserver with Master: 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757)
I20260926 14:52:56.262934 27967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:41887
I20260926 14:52:56.272806 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:52:56.281193 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.274596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.12.68.254:45237)
0926 14:52:56.274727 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:56.274732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:56.274883 (+ 151us) server_negotiation.cc:407] Connection header received
0926 14:52:56.274988 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:56.274991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:56.275031 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:56.275092 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:52:56.275738 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.276283 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.277072 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.277281 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.277363 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.278601 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:52:56.278604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:52:56.278606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:52:56.278627 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:52:56.279648 (+ 1021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:56.280081 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:56.280085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:56.280087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:56.280159 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:52:56.280393 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:52:56.280396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:52:56.280398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:52:56.280598 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:52:56.280729 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:56.280938 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 14:52:56.281043 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:52:56.283821 27967 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:52:56.286744 27967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:52:56.300079 28442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.295408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.12.68.195:33757)
0926 14:52:56.296165 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:56.296175 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:56.296195 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:52:56.296262 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:56.296267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:56.296429 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:56.296570 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:56.296741 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.297712 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.298924 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.299444 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.299537 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.299670 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:56.299826 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 14:52:56.299934 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":145,"threads_started":1}
I20260926 14:52:56.300071 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.295119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:33757 (local address 127.0.0.1:36606)
0926 14:52:56.295727 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:56.295739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:56.295816 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:56.296559 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:56.296562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:56.296580 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:56.296654 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:56.296660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.297823 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.297827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:56.298821 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:56.298831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.299552 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.299555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:56.299657 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.299693 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 14:52:56.299790 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 14:52:56.299934 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260926 14:52:56.300253 28437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.294423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:35281 (local address 127.0.0.1:46878)
0926 14:52:56.294846 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:56.294890 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:56.295002 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:56.296137 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:56.296140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:56.296155 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:56.296246 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:56.296252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.297566 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.297569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:56.298702 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:56.298712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.299825 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.299828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:56.299974 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.299993 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 14:52:56.300046 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 14:52:56.300114 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":38,"thread_start_us":77,"threads_started":1}
I20260926 14:52:56.300312 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.294727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.12.68.194:42911)
0926 14:52:56.295116 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:56.295127 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:56.295394 (+ 267us) server_negotiation.cc:407] Connection header received
0926 14:52:56.295475 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:56.295481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:56.295695 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:56.295840 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:56.296258 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.297138 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.298652 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.299455 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.299532 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.299685 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:56.300103 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 14:52:56.300191 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":92,"threads_started":1}
I20260926 14:52:56.300644 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.295071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46878 (local address 127.12.68.193:35281)
0926 14:52:56.295490 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:56.295500 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:56.295518 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:52:56.295579 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:56.295583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:56.295997 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:56.296149 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:56.296327 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.297438 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.298822 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.299580 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.299665 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.299777 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:56.300425 (+ 648us) server_negotiation.cc:299] Negotiation successful
0926 14:52:56.300510 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":120,"threads_started":1}
I20260926 14:52:56.300630 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.294556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:42911 (local address 127.0.0.1:45830)
0926 14:52:56.295218 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:56.295230 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:56.295308 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:56.295964 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:56.295970 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:56.295993 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:56.296130 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:56.296137 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.297264 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.297266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:56.298524 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:56.298534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.299777 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.299780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:56.299895 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.299972 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 14:52:56.300463 (+ 491us) client_negotiation.cc:240] Negotiation successful
0926 14:52:56.300526 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":35,"thread_start_us":95,"threads_started":1}
I20260926 14:52:56.303121 28360 tablet_service.cc:1467] Processing CreateTablet for tablet 7c329cca9f8d4a91821acf92b73cd69b (DEFAULT_TABLE table=test-table [id=357fcfd46bca4fbba234702f0a632697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:52:56.303121 28092 tablet_service.cc:1467] Processing CreateTablet for tablet 7c329cca9f8d4a91821acf92b73cd69b (DEFAULT_TABLE table=test-table [id=357fcfd46bca4fbba234702f0a632697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:52:56.304165 28092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c329cca9f8d4a91821acf92b73cd69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:56.304183 28360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c329cca9f8d4a91821acf92b73cd69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:56.304648 28226 tablet_service.cc:1467] Processing CreateTablet for tablet 7c329cca9f8d4a91821acf92b73cd69b (DEFAULT_TABLE table=test-table [id=357fcfd46bca4fbba234702f0a632697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:52:56.305536 28226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c329cca9f8d4a91821acf92b73cd69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:52:56.309818 28443 tablet_bootstrap.cc:492] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6: Bootstrap starting.
I20260926 14:52:56.311250 28444 tablet_bootstrap.cc:492] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Bootstrap starting.
I20260926 14:52:56.312445 28443 tablet_bootstrap.cc:654] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:56.312758 28445 tablet_bootstrap.cc:492] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: Bootstrap starting.
I20260926 14:52:56.313428 28444 tablet_bootstrap.cc:654] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:56.313434 28443 log.cc:824] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:56.314420 28444 log.cc:824] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:56.315079 28445 tablet_bootstrap.cc:654] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: Neither blocks nor log segments found. Creating new log.
I20260926 14:52:56.315763 28445 log.cc:824] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: Log is configured to *not* fsync() on all Append() calls
I20260926 14:52:56.316032 28443 tablet_bootstrap.cc:492] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6: No bootstrap required, opened a new log
I20260926 14:52:56.316213 28443 ts_tablet_manager.cc:1397] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 14:52:56.317528 28445 tablet_bootstrap.cc:492] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: No bootstrap required, opened a new log
I20260926 14:52:56.317689 28445 ts_tablet_manager.cc:1397] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 14:52:56.319075 28443 raft_consensus.cc:348] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } } peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } }
I20260926 14:52:56.319290 28443 raft_consensus.cc:374] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:56.319350 28443 raft_consensus.cc:729] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259cbbb3436140beb73e9ac00a6eb9e6, State: Initialized, Role: FOLLOWER
I20260926 14:52:56.319749 28443 consensus_queue.cc:260] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } } peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } }
I20260926 14:52:56.320389 28426 heartbeater.cc:503] Master 127.12.68.254:45237 was elected leader, sending a full tablet report...
I20260926 14:52:56.320434 28445 raft_consensus.cc:348] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } } peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } }
I20260926 14:52:56.320617 28443 ts_tablet_manager.cc:1428] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 14:52:56.320678 28445 raft_consensus.cc:374] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:56.320765 28445 raft_consensus.cc:729] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1bed65d37794758b8a05c6bf9196d60, State: Initialized, Role: FOLLOWER
I20260926 14:52:56.321110 28445 consensus_queue.cc:260] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } } peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } }
I20260926 14:52:56.321681 28292 heartbeater.cc:503] Master 127.12.68.254:45237 was elected leader, sending a full tablet report...
I20260926 14:52:56.321777 28444 tablet_bootstrap.cc:492] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: No bootstrap required, opened a new log
I20260926 14:52:56.321884 28445 ts_tablet_manager.cc:1428] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260926 14:52:56.321942 28444 ts_tablet_manager.cc:1397] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260926 14:52:56.325242 28444 raft_consensus.cc:348] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } } peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } }
I20260926 14:52:56.325497 28444 raft_consensus.cc:374] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:52:56.325560 28444 raft_consensus.cc:729] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69c45c6a4d946569810a68678df60a6, State: Initialized, Role: FOLLOWER
I20260926 14:52:56.325949 28444 consensus_queue.cc:260] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } } peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } }
I20260926 14:52:56.326490 28158 heartbeater.cc:503] Master 127.12.68.254:45237 was elected leader, sending a full tablet report...
I20260926 14:52:56.326643 28444 ts_tablet_manager.cc:1428] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:52:56.332077 28451 raft_consensus.cc:482] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:52:56.332211 28451 raft_consensus.cc:504] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } } peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } }
I20260926 14:52:56.333098 28451 leader_election.cc:288] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911), 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757)
I20260926 14:52:56.336412 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.333287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:42911 (local address 127.12.68.193:48445)
0926 14:52:56.333404 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:56.333420 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:56.333497 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:56.333732 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:56.333735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:56.333751 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:56.333855 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:56.333862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.334638 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.334641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:56.335387 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:56.335395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.336031 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.336034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:56.336137 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.336159 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:52:56.336224 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:52:56.336289 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 14:52:56.336412 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.333340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:48445 (local address 127.12.68.194:42911)
0926 14:52:56.333444 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:56.333448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:56.333461 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:52:56.333609 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:56.333612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:56.333655 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:56.333721 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:56.333946 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.334541 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.335497 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.335929 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.336031 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.336081 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:56.336250 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 14:52:56.336314 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:52:56.336843 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.333521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:33757 (local address 127.12.68.193:35535)
0926 14:52:56.333864 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:56.333877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:52:56.333999 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:52:56.334217 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:52:56.334219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:52:56.334234 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:52:56.334339 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:56.334345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.334980 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.334983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:56.335738 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:52:56.335746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.336387 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.336390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:52:56.336590 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.336610 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:52:56.336671 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:52:56.336741 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":90,"threads_started":1}
I20260926 14:52:56.336877 28442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.333604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:35535 (local address 127.12.68.195:33757)
0926 14:52:56.333713 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:56.333718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:56.333941 (+ 223us) server_negotiation.cc:407] Connection header received
0926 14:52:56.334087 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:56.334090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:56.334144 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:56.334209 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:52:56.334425 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.334889 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.335842 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.336282 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.336383 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.336427 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:56.336708 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 14:52:56.336776 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:52:56.336869 28246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c329cca9f8d4a91821acf92b73cd69b" candidate_uuid: "f69c45c6a4d946569810a68678df60a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1bed65d37794758b8a05c6bf9196d60" is_pre_election: true
I20260926 14:52:56.337121 28246 raft_consensus.cc:2454] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f69c45c6a4d946569810a68678df60a6 in term 0.
I20260926 14:52:56.337358 28380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c329cca9f8d4a91821acf92b73cd69b" candidate_uuid: "f69c45c6a4d946569810a68678df60a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" is_pre_election: true
I20260926 14:52:56.337551 28047 leader_election.cc:302] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1bed65d37794758b8a05c6bf9196d60, f69c45c6a4d946569810a68678df60a6; no voters:
I20260926 14:52:56.337599 28380 raft_consensus.cc:2454] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f69c45c6a4d946569810a68678df60a6 in term 0.
I20260926 14:52:56.337780 28451 raft_consensus.cc:2781] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:52:56.337857 28451 raft_consensus.cc:482] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:52:56.337905 28451 raft_consensus.cc:3037] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:56.338817 28451 raft_consensus.cc:504] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } } peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } }
I20260926 14:52:56.339078 28451 leader_election.cc:288] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 1 election: Requested vote from peers f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911), 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757)
I20260926 14:52:56.339378 28246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c329cca9f8d4a91821acf92b73cd69b" candidate_uuid: "f69c45c6a4d946569810a68678df60a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1bed65d37794758b8a05c6bf9196d60"
I20260926 14:52:56.339389 28380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c329cca9f8d4a91821acf92b73cd69b" candidate_uuid: "f69c45c6a4d946569810a68678df60a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259cbbb3436140beb73e9ac00a6eb9e6"
I20260926 14:52:56.339468 28246 raft_consensus.cc:3037] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:56.339504 28380 raft_consensus.cc:3037] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:52:56.340399 28246 raft_consensus.cc:2454] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f69c45c6a4d946569810a68678df60a6 in term 1.
I20260926 14:52:56.340507 28380 raft_consensus.cc:2454] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f69c45c6a4d946569810a68678df60a6 in term 1.
I20260926 14:52:56.340656 28047 leader_election.cc:302] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1bed65d37794758b8a05c6bf9196d60, f69c45c6a4d946569810a68678df60a6; no voters:
I20260926 14:52:56.340807 28451 raft_consensus.cc:2781] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:52:56.340999 28451 raft_consensus.cc:686] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 1 LEADER]: Becoming Leader. State: Replica: f69c45c6a4d946569810a68678df60a6, State: Running, Role: LEADER
I20260926 14:52:56.341285 28451 consensus_queue.cc:237] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } } peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } }
W20260926 14:52:56.341735 28159 tablet.cc:2369] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:52:56.344340 27967 catalog_manager.cc:5495] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f69c45c6a4d946569810a68678df60a6 (127.12.68.193). New cstate: current_term: 1 leader_uuid: "f69c45c6a4d946569810a68678df60a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } health_report { overall_health: UNKNOWN } } }
I20260926 14:52:56.361956 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.358756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46890 (local address 127.12.68.193:35281)
0926 14:52:56.358874 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:52:56.358879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:52:56.358893 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:52:56.358968 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:52:56.358971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:52:56.359016 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:52:56.359100 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:52:56.359327 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.359809 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.360643 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:52:56.360853 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:52:56.360936 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:52:56.361215 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:52:56.361583 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:52:56.361685 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:52:56.361775 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 14:52:56.361827 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:52:56.367518 28380 raft_consensus.cc:1261] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6 [term 1 FOLLOWER]: Refusing update from remote peer f69c45c6a4d946569810a68678df60a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:52:56.367540 28246 raft_consensus.cc:1261] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60 [term 1 FOLLOWER]: Refusing update from remote peer f69c45c6a4d946569810a68678df60a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:52:56.368168 28451 consensus_queue.cc:1035] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:52:56.368348 28453 consensus_queue.cc:1035] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:52:56.376263 28458 mvcc.cc:205] Tried to move back new op lower bound from 7333619205594652672 to 7333619205496647680. Current Snapshot: MvccSnapshot[applied={T|T < 7333619205594652672}]
I20260926 14:52:56.376883 28457 mvcc.cc:205] Tried to move back new op lower bound from 7333619205594652672 to 7333619205496647680. Current Snapshot: MvccSnapshot[applied={T|T < 7333619205594652672}]
I20260926 14:52:56.382740 27966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:52:56.382889 27966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:52:56.385155 27966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Sending DeleteTablet for 3 replicas of tablet 7c329cca9f8d4a91821acf92b73cd69b
I20260926 14:52:56.385896 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 28029
I20260926 14:52:56.385903 28226 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c329cca9f8d4a91821acf92b73cd69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:52:56 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:45830
I20260926 14:52:56.386047 28360 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c329cca9f8d4a91821acf92b73cd69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:52:56 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:36606
I20260926 14:52:56.387319 28466 tablet_replica.cc:330] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: stopping tablet replica
I20260926 14:52:56.387564 28466 raft_consensus.cc:2229] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:52:56.387814 28466 raft_consensus.cc:2258] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:52:56.388983 28466 ts_tablet_manager.cc:1905] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:52:56.391151 28467 tablet_replica.cc:330] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6: stopping tablet replica
I20260926 14:52:56.391182 28466 ts_tablet_manager.cc:1918] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:52:56.391302 28466 log.cc:1196] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/7c329cca9f8d4a91821acf92b73cd69b
W20260926 14:52:56.391299 27952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:35281 (error 108)
I20260926 14:52:56.391450 28467 raft_consensus.cc:2229] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:52:56.391495 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 28163
I20260926 14:52:56.391769 28467 raft_consensus.cc:2258] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:52:56.392782 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.392515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:35281 (local address 127.0.0.1:46900)
0926 14:52:56.392624 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:56.392693 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 14:52:56.392935 27952 catalog_manager.cc:4584] TS f69c45c6a4d946569810a68678df60a6 (127.12.68.193:35281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c329cca9f8d4a91821acf92b73cd69b: Network error: Client connection negotiation failed: client connection to 127.12.68.193:35281: connect: Connection refused (error 111)
I20260926 14:52:56.393421 28467 ts_tablet_manager.cc:1905] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:52:56.395341 28467 ts_tablet_manager.cc:1918] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:52:56.395419 28467 log.cc:1196] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/7c329cca9f8d4a91821acf92b73cd69b
I20260926 14:52:56.395690 28467 ts_tablet_manager.cc:1939] T 7c329cca9f8d4a91821acf92b73cd69b P 259cbbb3436140beb73e9ac00a6eb9e6: Deleting consensus metadata
I20260926 14:52:56.396703 27953 catalog_manager.cc:4841] TS 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757): tablet 7c329cca9f8d4a91821acf92b73cd69b (table test-table [id=357fcfd46bca4fbba234702f0a632697]) successfully deleted
I20260926 14:52:56.396911 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 28297
I20260926 14:52:56.397807 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:52:56.397580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:42911 (local address 127.0.0.1:45836)
0926 14:52:56.397671 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 14:52:56.397722 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.194:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 14:52:56.401784 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 27935
I20260926 14:53:16.407116 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:45237
--webserver_interface=127.12.68.254
--webserver_port=43873
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:45237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:16.508455 28469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:16.508675 28469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:16.508711 28469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:16.512552 28469 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 14:53:16.512591 28469 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 14:53:16.512606 28469 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 14:53:16.512619 28469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:53:16.512633 28469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:16.512651 28469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:16.512676 28469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:53:16.512704 28469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:53:16.517606 28469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:45237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:45237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=43873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:16.518452 28469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:16.519333 28469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:16.530635 28478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:16.530655 28475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:16.530655 28476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:16.530920 28469 server_base.cc:1034] running on GCE node
I20260926 14:53:16.531409 28469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:16.532435 28469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:16.533591 28469 hybrid_clock.cc:648] HybridClock initialized: now 1790434396533574 us; error 27 us; skew 500 ppm
Sep 26 14:53:16 dist-test-slave-24mn krb5kdc[27922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434396, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:16.536154 28469 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:53:16.537019 28469 webserver.cc:460] Webserver started at http://127.12.68.254:43873/ using document root <none> and password file <none>
I20260926 14:53:16.537490 28469 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:16.537560 28469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:16.541006 28469 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:53:16.542889 28485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:16.544008 28469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:53:16.544099 28469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "b999bf78bccc4b56af25a62df1afc8c9"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "62f7f6820ec4b62dac931243caf0bd2f"
server_key_iv: "9047f23f5331b547aec272c8b62ce9a3"
server_key_version: "encryptionkey@0"
I20260926 14:53:16.544612 28469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:16.561101 28469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:16.563643 28469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:16.563776 28469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:16.570055 28469 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:45237
I20260926 14:53:16.570289 28537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:45237 every 8 connection(s)
I20260926 14:53:16.571041 28469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:53:16.572604 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 28469
I20260926 14:53:16.573055 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:35281
--local_ip_for_outbound_sockets=127.12.68.193
--tserver_master_addrs=127.12.68.254:45237
--webserver_port=42411
--webserver_interface=127.12.68.193
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:53:16.575711 28538 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:53:16.577277 28538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Bootstrap starting.
I20260926 14:53:16.592980 28538 log.cc:824] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:16.600852 28538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:53:16.601147 28538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Bootstrap complete.
I20260926 14:53:16.604768 28538 raft_consensus.cc:348] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } }
I20260926 14:53:16.605127 28538 raft_consensus.cc:729] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b999bf78bccc4b56af25a62df1afc8c9, State: Initialized, Role: FOLLOWER
I20260926 14:53:16.605566 28538 consensus_queue.cc:260] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } }
I20260926 14:53:16.605672 28538 raft_consensus.cc:388] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:53:16.605734 28538 raft_consensus.cc:482] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:53:16.605818 28538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:53:16.607573 28538 raft_consensus.cc:504] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } }
I20260926 14:53:16.607860 28538 leader_election.cc:302] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b999bf78bccc4b56af25a62df1afc8c9; no voters:
I20260926 14:53:16.608148 28538 leader_election.cc:288] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:53:16.608232 28542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:53:16.608422 28542 raft_consensus.cc:686] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 2 LEADER]: Becoming Leader. State: Replica: b999bf78bccc4b56af25a62df1afc8c9, State: Running, Role: LEADER
I20260926 14:53:16.608742 28542 consensus_queue.cc:237] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } }
I20260926 14:53:16.609293 28538 sys_catalog.cc:564] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:53:16.609607 28542 sys_catalog.cc:455] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b999bf78bccc4b56af25a62df1afc8c9. Latest consensus state: current_term: 2 leader_uuid: "b999bf78bccc4b56af25a62df1afc8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } } }
I20260926 14:53:16.609789 28542 sys_catalog.cc:458] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:16.609929 28543 sys_catalog.cc:455] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b999bf78bccc4b56af25a62df1afc8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } } }
I20260926 14:53:16.610106 28543 sys_catalog.cc:458] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:16.610196 28546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:53:16.614603 28546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:53:16.615245 28546 catalog_manager.cc:1259] Loaded cluster ID: c05b244356ea4c8daa63becf549718a3
I20260926 14:53:16.615306 28546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:53:16.617704 28546 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:53:16.618384 28546 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Loaded TSK: 0
I20260926 14:53:16.632344 28546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Generated new TSK 1
I20260926 14:53:16.632558 28546 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:53:16.678408 28540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:16.678647 28540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:16.678705 28540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:16.682597 28540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:16.682646 28540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:16.682719 28540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:53:16.687635 28540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:35281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=42411
--tserver_master_addrs=127.12.68.254:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:16.688586 28540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:16.689481 28540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:16.698045 28565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:16.698133 28566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:16.698067 28568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:16.698546 28540 server_base.cc:1034] running on GCE node
I20260926 14:53:16.698961 28540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:16.699640 28540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:16.700829 28540 hybrid_clock.cc:648] HybridClock initialized: now 1790434396700782 us; error 48 us; skew 500 ppm
Sep 26 14:53:16 dist-test-slave-24mn krb5kdc[27922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434396, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:16.703909 28540 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:53:16.704993 28540 webserver.cc:460] Webserver started at http://127.12.68.193:42411/ using document root <none> and password file <none>
I20260926 14:53:16.705626 28540 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:16.705721 28540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:16.710501 28540 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:53:16.712920 28575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:16.714277 28540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:53:16.714391 28540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "f69c45c6a4d946569810a68678df60a6"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "55cc0d15105a14b142b4bd6bd67dd96e"
server_key_iv: "d3e6c17ed933429f5917bbcc2c58ddea"
server_key_version: "encryptionkey@0"
I20260926 14:53:16.715035 28540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:16.742866 28540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:16.746037 28540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:16.746214 28540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:16.746852 28540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:16.748026 28582 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:53:16.751750 28540 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:53:16.751816 28540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260926 14:53:16.751873 28540 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:53:16.753815 28540 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:53:16.753863 28540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 14:53:16.754160 28582 tablet_bootstrap.cc:492] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Bootstrap starting.
I20260926 14:53:16.763116 28540 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:35281
I20260926 14:53:16.763360 28689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:35281 every 8 connection(s)
I20260926 14:53:16.764194 28540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:53:16.769369 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 28540
I20260926 14:53:16.769744 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:42911
--local_ip_for_outbound_sockets=127.12.68.194
--tserver_master_addrs=127.12.68.254:45237
--webserver_port=35533
--webserver_interface=127.12.68.194
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:53:16.773478 28582 log.cc:824] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:53:16 dist-test-slave-24mn krb5kdc[27922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434396, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:53:16.780283 28692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:16.766218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:40531 (local address 127.12.68.254:45237)
0926 14:53:16.766577 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:16.766595 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:16.766938 (+ 343us) server_negotiation.cc:407] Connection header received
0926 14:53:16.768269 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:16.768279 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:16.768548 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:16.768790 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:16.770290 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:16.771563 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:16.772679 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:16.773316 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:16.773468 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:16.775865 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:16.775870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:16.775902 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:16.775940 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:16.777425 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:16.777880 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:16.777885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:16.777889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:16.778004 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:16.778301 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:16.778304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:16.778306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:16.778611 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:16.778727 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:16.779216 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 14:53:16.779449 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":112,"threads_started":1}
I20260926 14:53:16.780293 28693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:16.766193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:40531)
0926 14:53:16.766759 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:16.766812 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:16.767910 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:16.768932 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:16.768941 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:16.769385 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:16.770109 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:16.770129 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:16.771705 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:16.771708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:16.772556 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:16.772573 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:16.773459 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:16.773463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:16.773645 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:16.774334 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:16.774353 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:16.775695 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:16.777532 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:16.777537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:16.777549 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:16.777779 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:16.778090 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:16.778093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:16.778095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:16.778220 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:16.778717 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:16.778722 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:16.778942 (+ 220us) client_negotiation.cc:769] Sending connection context
0926 14:53:16.779091 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 14:53:16.779392 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":111,"threads_started":1}
I20260926 14:53:16.781872 28582 tablet_bootstrap.cc:492] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:53:16.782249 28582 tablet_bootstrap.cc:492] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Bootstrap complete.
I20260926 14:53:16.782541 28690 heartbeater.cc:347] Connected to a master server at 127.12.68.254:45237
I20260926 14:53:16.782745 28690 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:16.783166 28690 heartbeater.cc:511] Master 127.12.68.254:45237 requested a full tablet report, sending...
I20260926 14:53:16.783529 28582 ts_tablet_manager.cc:1397] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.005s
I20260926 14:53:16.785347 28502 ts_manager.cc:194] Registered new tserver with Master: f69c45c6a4d946569810a68678df60a6 (127.12.68.193:35281)
I20260926 14:53:16.786268 28582 raft_consensus.cc:348] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } } peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } }
I20260926 14:53:16.786746 28582 raft_consensus.cc:729] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69c45c6a4d946569810a68678df60a6, State: Initialized, Role: FOLLOWER
I20260926 14:53:16.787230 28582 consensus_queue.cc:260] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } } peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } }
I20260926 14:53:16.788218 28582 ts_tablet_manager.cc:1428] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:53:16.789605 28502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:40531
I20260926 14:53:16.792183 28690 heartbeater.cc:503] Master 127.12.68.254:45237 was elected leader, sending a full tablet report...
Sep 26 14:53:16 dist-test-slave-24mn krb5kdc[27922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434396, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for kudu/127.12.68.193@KRBTEST.COM
I20260926 14:53:16.801649 28697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:16.787966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:35281 (local address 127.0.0.1:58746)
0926 14:53:16.788274 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:16.788338 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:16.788458 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:16.789200 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:16.789206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:16.789590 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:16.789705 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:16.789712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:16.790871 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:16.790874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:16.791509 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:16.791518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:16.795423 (+ 3905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:16.795427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:16.795598 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:16.796188 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:16.796203 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:16.797753 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:16.799828 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:16.799836 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:16.799841 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:16.800154 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:16.800464 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:16.800467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:16.800469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:16.800556 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:16.801045 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:16.801051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:16.801182 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 14:53:16.801316 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 14:53:16.801485 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
I20260926 14:53:16.801672 28698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:16.788538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.12.68.193:35281)
0926 14:53:16.788778 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:16.788788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:16.788807 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:53:16.788861 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:16.788865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:16.789044 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:16.789171 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:16.789810 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:16.790773 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:16.791614 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:16.791988 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:16.792077 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:16.797904 (+ 5827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:16.797924 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:16.797937 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:16.797963 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:16.799714 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:16.800265 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:16.800271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:16.800273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:16.800360 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:16.800657 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:16.800660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:16.800662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:16.800940 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:16.801063 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:16.801412 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 14:53:16.801559 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260926 14:53:16.802539 28624 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c329cca9f8d4a91821acf92b73cd69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:52:56 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:58746
I20260926 14:53:16.803267 28699 tablet_replica.cc:330] stopping tablet replica
I20260926 14:53:16.803530 28699 raft_consensus.cc:2229] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:53:16.803742 28699 raft_consensus.cc:2258] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:16.804610 28699 ts_tablet_manager.cc:1905] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:16.808010 28699 ts_tablet_manager.cc:1918] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:53:16.808086 28699 log.cc:1196] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal/wals/7c329cca9f8d4a91821acf92b73cd69b
I20260926 14:53:16.808303 28699 ts_tablet_manager.cc:1939] T 7c329cca9f8d4a91821acf92b73cd69b P f69c45c6a4d946569810a68678df60a6: Deleting consensus metadata
I20260926 14:53:16.810189 28487 catalog_manager.cc:4841] TS f69c45c6a4d946569810a68678df60a6 (127.12.68.193:35281): tablet 7c329cca9f8d4a91821acf92b73cd69b (table test-table [id=357fcfd46bca4fbba234702f0a632697]) successfully deleted
W20260926 14:53:16.880710 28694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:16.880915 28694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:16.880970 28694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:16.884801 28694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:16.884847 28694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:16.884922 28694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:53:16.889765 28694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:42911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=35533
--tserver_master_addrs=127.12.68.254:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:16.890617 28694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:16.891469 28694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:16.899993 28704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:16.900127 28694 server_base.cc:1034] running on GCE node
W20260926 14:53:16.899977 28705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:16.899961 28707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:16.900522 28694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:16.901046 28694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:16.902212 28694 hybrid_clock.cc:648] HybridClock initialized: now 1790434396902194 us; error 27 us; skew 500 ppm
Sep 26 14:53:16 dist-test-slave-24mn krb5kdc[27922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434396, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:16.904673 28694 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:53:16.905436 28694 webserver.cc:460] Webserver started at http://127.12.68.194:35533/ using document root <none> and password file <none>
I20260926 14:53:16.905939 28694 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:16.906011 28694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:16.909483 28694 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:53:16.911444 28714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:16.912546 28694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:53:16.912647 28694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "f1bed65d37794758b8a05c6bf9196d60"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "f628e21e37a9bcf4f2966b8162850f33"
server_key_iv: "0a47472a4f95d6d57ad9d0a0ed9fc6c0"
server_key_version: "encryptionkey@0"
I20260926 14:53:16.913113 28694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:16.926079 28694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:16.928474 28694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:16.928612 28694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:16.929085 28694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:16.929971 28721 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 14:53:16.933223 28721 ts_tablet_manager.cc:1852] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 14:53:16.933580 28721 ts_tablet_manager.cc:1905] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:16.937037 28721 ts_tablet_manager.cc:1918] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:53:16.937120 28721 log.cc:1196] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/7c329cca9f8d4a91821acf92b73cd69b
I20260926 14:53:16.937350 28721 ts_tablet_manager.cc:1939] T 7c329cca9f8d4a91821acf92b73cd69b P f1bed65d37794758b8a05c6bf9196d60: Deleting consensus metadata
I20260926 14:53:16.937614 28694 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 14:53:16.937670 28694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260926 14:53:16.937719 28694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:16.937748 28694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:16.945851 28694 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:42911
I20260926 14:53:16.945887 28828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:42911 every 8 connection(s)
I20260926 14:53:16.946595 28694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:53:16 dist-test-slave-24mn krb5kdc[27922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434396, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:53:16.956499 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 28694
I20260926 14:53:16.956940 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:33757
--local_ip_for_outbound_sockets=127.12.68.195
--tserver_master_addrs=127.12.68.254:45237
--webserver_port=40273
--webserver_interface=127.12.68.195
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:53:16.958043 28692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:16.948133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:47595 (local address 127.12.68.254:45237)
0926 14:53:16.948254 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:16.948258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:16.949054 (+ 796us) server_negotiation.cc:407] Connection header received
0926 14:53:16.949894 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:16.949897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:16.949958 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:16.950037 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:16.951099 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:16.951616 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:16.952694 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:16.952925 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:16.953016 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:16.955086 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:16.955089 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:16.955090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:16.955113 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:16.956217 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:16.956690 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:16.956694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:16.956696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:16.956789 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:16.957067 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:16.957070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:16.957072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:16.957238 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:16.957321 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:16.957756 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 14:53:16.957879 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:53:16.958873 28831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:16.948516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:47595)
0926 14:53:16.948922 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:16.948959 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:16.949703 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:16.950191 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:16.950197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:16.950537 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:16.950962 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:16.950974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:16.951725 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:16.951728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:16.952583 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:16.952593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:16.953048 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:16.953052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:16.953205 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:16.953671 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:16.953686 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:16.954936 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:16.956328 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:16.956333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:16.956345 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:16.956581 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:16.956867 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:16.956869 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:16.956870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:16.956979 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:16.957330 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:16.957336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:16.957551 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 14:53:16.957700 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 14:53:16.957973 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":101,"threads_started":1}
I20260926 14:53:16.959901 28829 heartbeater.cc:347] Connected to a master server at 127.12.68.254:45237
I20260926 14:53:16.960093 28829 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:16.960485 28829 heartbeater.cc:511] Master 127.12.68.254:45237 requested a full tablet report, sending...
I20260926 14:53:16.961426 28502 ts_manager.cc:194] Registered new tserver with Master: f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911)
I20260926 14:53:16.961939 28502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:47595
W20260926 14:53:17.060400 28832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:17.060604 28832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:17.060649 28832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:17.064518 28832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:17.064561 28832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:17.064623 28832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:53:17.069468 28832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:33757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=40273
--tserver_master_addrs=127.12.68.254:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:17.070313 28832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:17.071201 28832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:17.079752 28840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:17.079788 28837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:17.079991 28832 server_base.cc:1034] running on GCE node
W20260926 14:53:17.079793 28838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:17.080435 28832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:17.081000 28832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:17.082155 28832 hybrid_clock.cc:648] HybridClock initialized: now 1790434397082131 us; error 37 us; skew 500 ppm
Sep 26 14:53:17 dist-test-slave-24mn krb5kdc[27922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434397, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:17.084510 28832 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:53:17.085247 28832 webserver.cc:460] Webserver started at http://127.12.68.195:40273/ using document root <none> and password file <none>
I20260926 14:53:17.085719 28832 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:17.085791 28832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:17.089179 28832 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:53:17.090921 28847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:17.091982 28832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:53:17.092084 28832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "259cbbb3436140beb73e9ac00a6eb9e6"
format_stamp: "Formatted at 2026-09-26 14:52:56 on dist-test-slave-24mn"
server_key: "d6386eb578f33ac3f363874c7865ab62"
server_key_iv: "0368d0b053cf65ce5fe8d6b472db736a"
server_key_version: "encryptionkey@0"
I20260926 14:53:17.092582 28832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:17.112695 28832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:17.115082 28832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:17.115240 28832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:17.115727 28832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:17.116484 28832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:17.116535 28832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:17.116592 28832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:17.116618 28832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:17.124405 28832 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:33757
I20260926 14:53:17.124459 28960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:33757 every 8 connection(s)
I20260926 14:53:17.125262 28832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:53:17.132833 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 28832
Sep 26 14:53:17 dist-test-slave-24mn krb5kdc[27922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434397, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:53:17.137290 28692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.126786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:34989 (local address 127.12.68.254:45237)
0926 14:53:17.126900 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:17.126904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:17.127683 (+ 779us) server_negotiation.cc:407] Connection header received
0926 14:53:17.128856 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:17.128858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:17.128902 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:17.128967 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:17.129955 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.130462 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.131517 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.131770 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.131850 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:17.134104 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:17.134108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:17.134111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:17.134138 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:17.135447 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:17.135874 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:17.135877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:17.135899 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:17.135981 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:17.136245 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:17.136248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:17.136249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:17.136450 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:17.136518 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:17.137031 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 14:53:17.137130 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":40}
I20260926 14:53:17.138093 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.127131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:34989)
0926 14:53:17.127514 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:17.127566 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:17.128687 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:17.129103 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:17.129109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:17.129419 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:17.129854 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:17.129866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.130558 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.130560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:17.131401 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:17.131410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.131899 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.131904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:17.132047 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:17.132487 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:17.132501 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:17.133940 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:17.135551 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:17.135555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:17.135565 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:17.135790 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:17.136055 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:17.136058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:17.136059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:17.136173 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:17.136549 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:17.136555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:17.136821 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 14:53:17.136967 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 14:53:17.137251 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":102,"threads_started":1}
I20260926 14:53:17.139106 28961 heartbeater.cc:347] Connected to a master server at 127.12.68.254:45237
I20260926 14:53:17.139326 28961 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:17.139832 28961 heartbeater.cc:511] Master 127.12.68.254:45237 requested a full tablet report, sending...
I20260926 14:53:17.140825 28502 ts_manager.cc:194] Registered new tserver with Master: 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757)
I20260926 14:53:17.141125 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.133447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.12.68.254:45237)
0926 14:53:17.133659 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:17.133664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:17.133788 (+ 124us) server_negotiation.cc:407] Connection header received
0926 14:53:17.133879 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:17.133882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:17.133926 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:17.134027 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:17.134827 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.135396 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.136378 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.136629 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.136720 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:17.137964 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:17.137968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:17.137970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:17.137993 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:17.139534 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:17.139946 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:17.139950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:17.139951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:17.140031 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:17.140263 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:17.140266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:17.140267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:17.140441 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:17.140577 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:17.140877 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 14:53:17.140980 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":60,"threads_started":1}
I20260926 14:53:17.141412 28502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:34989
I20260926 14:53:17.142658 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:53:17.145800 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.143166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.12.68.254:45237)
0926 14:53:17.143256 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:17.143259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:17.143393 (+ 134us) server_negotiation.cc:407] Connection header received
0926 14:53:17.143495 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:17.143497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:17.143568 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:17.143641 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:53:17.143784 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.144293 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.145012 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.145213 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.145275 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:17.145466 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:53:17.145566 (+ 100us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 14:53:17.145727 (+ 161us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34494: token expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260926 14:53:17.145843 28965 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34494: token expired
W20260926 14:53:17.146142 28431 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.68.254:45237)
I20260926 14:53:17.152379 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.146431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.12.68.254:45237)
0926 14:53:17.146524 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:17.146528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:17.146540 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:53:17.146656 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:17.146659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:17.146692 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:17.146749 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:17.147285 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.147753 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.148512 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.148723 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.148800 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:17.149841 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:17.149844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:17.149847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:17.149869 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:17.150960 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:17.151357 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:17.151360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:17.151362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:17.151435 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:17.151669 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:17.151672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:17.151674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:17.151822 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:17.151942 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:17.152164 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 14:53:17.152258 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:53:17.154342 28502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:53:17.157380 28502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:53:17.164454 28624 tablet_service.cc:1467] Processing CreateTablet for tablet de203d0df61842f68d79eb3c11511c7d (DEFAULT_TABLE table=test-table [id=e530da9a5b954c59b11cdd412b775d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:17.165083 28624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de203d0df61842f68d79eb3c11511c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:17.167860 28582 tablet_bootstrap.cc:492] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6: Bootstrap starting.
I20260926 14:53:17.168459 28971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.163611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:33757 (local address 127.0.0.1:44956)
0926 14:53:17.164142 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:17.164156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:17.164236 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:17.164891 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:17.164894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:17.164912 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:17.164989 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:17.164994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.166163 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.166166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:17.167107 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:17.167116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.168046 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.168049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:17.168135 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:17.168176 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 14:53:17.168249 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:53:17.168317 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":39,"thread_start_us":104,"threads_started":1}
I20260926 14:53:17.168535 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.164045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.12.68.195:33757)
0926 14:53:17.164469 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:17.164480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:17.164502 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:53:17.164572 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:17.164577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:17.164755 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:17.164912 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:17.165075 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.166049 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.167213 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.167875 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.168045 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:17.168184 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:17.168299 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 14:53:17.168392 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":141,"threads_started":1}
I20260926 14:53:17.168566 28697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.163289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:42911 (local address 127.0.0.1:59042)
0926 14:53:17.163443 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:17.163456 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:17.163524 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:17.164349 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:17.164351 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:17.164367 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:17.164439 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:17.164443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.165685 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.165688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:17.166871 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:17.166881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.168261 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.168264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:17.168351 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:17.168372 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:53:17.168416 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 14:53:17.168476 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":134}
I20260926 14:53:17.168725 28582 tablet_bootstrap.cc:654] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:17.169137 28970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.163504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59042 (local address 127.12.68.194:42911)
0926 14:53:17.163804 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:17.163820 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:17.163840 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:53:17.163964 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:17.163970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:17.164185 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:17.164562 (+ 377us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:17.164717 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.165560 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.167011 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.167845 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.168703 (+ 858us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:17.168835 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:17.168900 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 14:53:17.168986 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":116,"threads_started":1}
I20260926 14:53:17.170226 28582 tablet_bootstrap.cc:492] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6: No bootstrap required, opened a new log
I20260926 14:53:17.170315 28582 ts_tablet_manager.cc:1397] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 14:53:17.170755 28582 raft_consensus.cc:348] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:17.170871 28582 raft_consensus.cc:374] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:17.170908 28582 raft_consensus.cc:729] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69c45c6a4d946569810a68678df60a6, State: Initialized, Role: FOLLOWER
I20260926 14:53:17.170500 28763 tablet_service.cc:1467] Processing CreateTablet for tablet de203d0df61842f68d79eb3c11511c7d (DEFAULT_TABLE table=test-table [id=e530da9a5b954c59b11cdd412b775d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:17.171030 28582 consensus_queue.cc:260] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:17.171279 28582 ts_tablet_manager.cc:1428] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:53:17.171491 28763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de203d0df61842f68d79eb3c11511c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:17.171537 28895 tablet_service.cc:1467] Processing CreateTablet for tablet de203d0df61842f68d79eb3c11511c7d (DEFAULT_TABLE table=test-table [id=e530da9a5b954c59b11cdd412b775d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:17.172504 28895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de203d0df61842f68d79eb3c11511c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:17.175791 28721 tablet_bootstrap.cc:492] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60: Bootstrap starting.
I20260926 14:53:17.177548 28721 tablet_bootstrap.cc:654] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:17.178200 28721 log.cc:824] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:17.178951 28975 tablet_bootstrap.cc:492] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6: Bootstrap starting.
I20260926 14:53:17.180039 28721 tablet_bootstrap.cc:492] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60: No bootstrap required, opened a new log
I20260926 14:53:17.180193 28721 ts_tablet_manager.cc:1397] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:53:17.181123 28975 tablet_bootstrap.cc:654] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:17.181748 28975 log.cc:824] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:17.182854 28721 raft_consensus.cc:348] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:17.183041 28721 raft_consensus.cc:374] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:17.183099 28721 raft_consensus.cc:729] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1bed65d37794758b8a05c6bf9196d60, State: Initialized, Role: FOLLOWER
I20260926 14:53:17.183236 28975 tablet_bootstrap.cc:492] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6: No bootstrap required, opened a new log
I20260926 14:53:17.183406 28975 ts_tablet_manager.cc:1397] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:53:17.183490 28721 consensus_queue.cc:260] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:17.184278 28829 heartbeater.cc:503] Master 127.12.68.254:45237 was elected leader, sending a full tablet report...
I20260926 14:53:17.184427 28721 ts_tablet_manager.cc:1428] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 14:53:17.186437 28975 raft_consensus.cc:348] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:17.186647 28975 raft_consensus.cc:374] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:17.186700 28975 raft_consensus.cc:729] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259cbbb3436140beb73e9ac00a6eb9e6, State: Initialized, Role: FOLLOWER
I20260926 14:53:17.187057 28975 consensus_queue.cc:260] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:17.187583 28961 heartbeater.cc:503] Master 127.12.68.254:45237 was elected leader, sending a full tablet report...
I20260926 14:53:17.187801 28975 ts_tablet_manager.cc:1428] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260926 14:53:17.197315 28830 tablet.cc:2369] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:53:17.239500 28978 raft_consensus.cc:482] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:53:17.239672 28978 raft_consensus.cc:504] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:17.240743 28978 leader_election.cc:288] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911), f69c45c6a4d946569810a68678df60a6 (127.12.68.193:35281)
I20260926 14:53:17.244437 28970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.241201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:41181 (local address 127.12.68.194:42911)
0926 14:53:17.241309 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:17.241313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:17.241326 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:53:17.241377 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:17.241380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:17.241423 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:17.241505 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:17.241745 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.242251 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.243493 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.243929 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.244037 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:17.244090 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:17.244244 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 14:53:17.244314 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:53:17.244443 28698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.241261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:38381 (local address 127.12.68.193:35281)
0926 14:53:17.241367 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:17.241370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:17.241382 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:53:17.241424 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:17.241426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:17.241474 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:17.241553 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:17.241799 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.242312 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.243501 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.243968 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.244076 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:17.244220 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:17.244262 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 14:53:17.244329 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:53:17.244443 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.240933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:42911 (local address 127.12.68.195:41181)
0926 14:53:17.241050 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:17.241064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:17.241209 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:17.241520 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:17.241523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:17.241539 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:17.241649 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:17.241655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.242355 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.242358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:17.243376 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:17.243385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.244033 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.244037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:17.244130 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:17.244151 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:53:17.244218 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:53:17.244285 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":54}
I20260926 14:53:17.244467 28979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.240958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:35281 (local address 127.12.68.195:38381)
0926 14:53:17.241214 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:17.241227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:17.241317 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:17.241577 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:17.241580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:17.241596 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:17.241700 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:17.241705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.242403 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.242406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:17.243388 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:17.243397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.244086 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.244089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:17.244179 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:17.244196 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 14:53:17.244252 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:53:17.244314 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260926 14:53:17.245148 28783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de203d0df61842f68d79eb3c11511c7d" candidate_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1bed65d37794758b8a05c6bf9196d60" is_pre_election: true
I20260926 14:53:17.245137 28644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de203d0df61842f68d79eb3c11511c7d" candidate_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f69c45c6a4d946569810a68678df60a6" is_pre_election: true
I20260926 14:53:17.245415 28644 raft_consensus.cc:2454] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259cbbb3436140beb73e9ac00a6eb9e6 in term 0.
I20260926 14:53:17.245415 28783 raft_consensus.cc:2454] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259cbbb3436140beb73e9ac00a6eb9e6 in term 0.
I20260926 14:53:17.245824 28850 leader_election.cc:302] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259cbbb3436140beb73e9ac00a6eb9e6, f1bed65d37794758b8a05c6bf9196d60; no voters:
I20260926 14:53:17.246040 28978 raft_consensus.cc:2781] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:53:17.246119 28978 raft_consensus.cc:482] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:53:17.246168 28978 raft_consensus.cc:3037] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:17.247119 28978 raft_consensus.cc:504] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:17.247412 28978 leader_election.cc:288] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [CANDIDATE]: Term 1 election: Requested vote from peers f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911), f69c45c6a4d946569810a68678df60a6 (127.12.68.193:35281)
I20260926 14:53:17.247701 28783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de203d0df61842f68d79eb3c11511c7d" candidate_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1bed65d37794758b8a05c6bf9196d60"
I20260926 14:53:17.247701 28644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de203d0df61842f68d79eb3c11511c7d" candidate_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f69c45c6a4d946569810a68678df60a6"
I20260926 14:53:17.247815 28644 raft_consensus.cc:3037] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:17.247831 28783 raft_consensus.cc:3037] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:17.249003 28783 raft_consensus.cc:2454] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259cbbb3436140beb73e9ac00a6eb9e6 in term 1.
I20260926 14:53:17.249003 28644 raft_consensus.cc:2454] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259cbbb3436140beb73e9ac00a6eb9e6 in term 1.
I20260926 14:53:17.249287 28849 leader_election.cc:302] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 259cbbb3436140beb73e9ac00a6eb9e6, f1bed65d37794758b8a05c6bf9196d60, f69c45c6a4d946569810a68678df60a6; no voters:
I20260926 14:53:17.249444 28978 raft_consensus.cc:2781] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:53:17.249643 28978 raft_consensus.cc:686] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [term 1 LEADER]: Becoming Leader. State: Replica: 259cbbb3436140beb73e9ac00a6eb9e6, State: Running, Role: LEADER
I20260926 14:53:17.249945 28978 consensus_queue.cc:237] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:17.252784 28502 catalog_manager.cc:5495] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195). New cstate: current_term: 1 leader_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } health_report { overall_health: UNKNOWN } } }
W20260926 14:53:17.265408 28691 tablet.cc:2369] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:53:17.269241 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.265975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.12.68.195:33757)
0926 14:53:17.266097 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:17.266102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:17.266115 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:53:17.266277 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:17.266279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:17.266327 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:17.266410 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:53:17.266631 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.267108 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.267922 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:17.268134 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:17.268228 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:17.268495 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:53:17.268871 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:53:17.268969 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:17.269047 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 14:53:17.269101 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:53:17.274761 28644 raft_consensus.cc:1261] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Refusing update from remote peer 259cbbb3436140beb73e9ac00a6eb9e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:53:17.274765 28783 raft_consensus.cc:1261] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60 [term 1 FOLLOWER]: Refusing update from remote peer 259cbbb3436140beb73e9ac00a6eb9e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:53:17.275341 28978 consensus_queue.cc:1035] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:53:17.275496 28980 consensus_queue.cc:1035] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:53:17.280159 28983 mvcc.cc:205] Tried to move back new op lower bound from 7333619291230892032 to 7333619291138236416. Current Snapshot: MvccSnapshot[applied={T|T < 7333619291230892032}]
I20260926 14:53:17.281242 28984 mvcc.cc:205] Tried to move back new op lower bound from 7333619291230892032 to 7333619291138236416. Current Snapshot: MvccSnapshot[applied={T|T < 7333619291230892032}]
I20260926 14:53:17.289491 28502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:53:17.289654 28502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:53:17.291842 28502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Sending DeleteTablet for 3 replicas of tablet de203d0df61842f68d79eb3c11511c7d
I20260926 14:53:17.292495 28624 tablet_service.cc:1514] Processing DeleteTablet for tablet de203d0df61842f68d79eb3c11511c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:17 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:58746
I20260926 14:53:17.292439 28763 tablet_service.cc:1514] Processing DeleteTablet for tablet de203d0df61842f68d79eb3c11511c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:17 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:59042
I20260926 14:53:17.292579 28895 tablet_service.cc:1514] Processing DeleteTablet for tablet de203d0df61842f68d79eb3c11511c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:17 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:44956
I20260926 14:53:17.292651 28699 tablet_replica.cc:330] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6: stopping tablet replica
I20260926 14:53:17.292752 28699 raft_consensus.cc:2229] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:53:17.292870 28699 raft_consensus.cc:2258] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:17.293169 28992 tablet_replica.cc:330] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60: stopping tablet replica
I20260926 14:53:17.293308 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 28540
I20260926 14:53:17.293634 28699 ts_tablet_manager.cc:1905] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:17.294050 28992 raft_consensus.cc:2229] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:53:17.294355 28993 tablet_replica.cc:330] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6: stopping tablet replica
I20260926 14:53:17.294376 28992 raft_consensus.cc:2258] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:17.294615 28993 raft_consensus.cc:2229] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:53:17.294960 28993 raft_consensus.cc:2258] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:17.295941 28992 ts_tablet_manager.cc:1905] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:17.296816 28993 ts_tablet_manager.cc:1905] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:17.298760 28992 ts_tablet_manager.cc:1918] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:53:17.298880 28992 log.cc:1196] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/de203d0df61842f68d79eb3c11511c7d
I20260926 14:53:17.299187 28992 ts_tablet_manager.cc:1939] T de203d0df61842f68d79eb3c11511c7d P f1bed65d37794758b8a05c6bf9196d60: Deleting consensus metadata
I20260926 14:53:17.299340 28993 ts_tablet_manager.cc:1918] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:53:17.299427 28993 log.cc:1196] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/de203d0df61842f68d79eb3c11511c7d
I20260926 14:53:17.299718 28993 ts_tablet_manager.cc:1939] T de203d0df61842f68d79eb3c11511c7d P 259cbbb3436140beb73e9ac00a6eb9e6: Deleting consensus metadata
I20260926 14:53:17.300510 28488 catalog_manager.cc:4841] TS f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911): tablet de203d0df61842f68d79eb3c11511c7d (table test-table [id=e530da9a5b954c59b11cdd412b775d51]) successfully deleted
I20260926 14:53:17.300684 28488 catalog_manager.cc:4841] TS 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757): tablet de203d0df61842f68d79eb3c11511c7d (table test-table [id=e530da9a5b954c59b11cdd412b775d51]) successfully deleted
W20260926 14:53:17.300690 28487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:35281 (error 108)
I20260926 14:53:17.301318 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 28694
I20260926 14:53:17.301573 28697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:17.301325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:35281 (local address 127.0.0.1:58760)
0926 14:53:17.301435 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:17.301501 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 14:53:17.301848 28487 catalog_manager.cc:4584] TS f69c45c6a4d946569810a68678df60a6 (127.12.68.193:35281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de203d0df61842f68d79eb3c11511c7d: Network error: Client connection negotiation failed: client connection to 127.12.68.193:35281: connect: Connection refused (error 111)
I20260926 14:53:17.306564 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 28832
I20260926 14:53:17.311651 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 28469
I20260926 14:53:37.316820 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:45237
--webserver_interface=127.12.68.254
--webserver_port=43873
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:45237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:37.417474 28995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:37.417686 28995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:37.417735 28995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:37.421574 28995 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 14:53:37.421615 28995 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 14:53:37.421636 28995 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 14:53:37.421653 28995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:53:37.421675 28995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:37.421707 28995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:37.421736 28995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:53:37.421763 28995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:53:37.426635 28995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:45237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:45237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=43873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:37.427481 28995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:37.428370 28995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:37.436203 29003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:37.436187 29000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:37.436295 28995 server_base.cc:1034] running on GCE node
W20260926 14:53:37.436182 29001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:37.437215 28995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:37.438187 28995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:37.439352 28995 hybrid_clock.cc:648] HybridClock initialized: now 1790434417439326 us; error 39 us; skew 500 ppm
Sep 26 14:53:37 dist-test-slave-24mn krb5kdc[27922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434417, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:37.441917 28995 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:53:37.442691 28995 webserver.cc:460] Webserver started at http://127.12.68.254:43873/ using document root <none> and password file <none>
I20260926 14:53:37.443185 28995 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:37.443244 28995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:37.446825 28995 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:53:37.448786 29010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:37.449918 28995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:53:37.450021 28995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "b999bf78bccc4b56af25a62df1afc8c9"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "62f7f6820ec4b62dac931243caf0bd2f"
server_key_iv: "9047f23f5331b547aec272c8b62ce9a3"
server_key_version: "encryptionkey@0"
I20260926 14:53:37.450500 28995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:37.462122 28995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:37.464550 28995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:37.464708 28995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:37.470870 28995 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:45237
I20260926 14:53:37.471101 29062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:45237 every 8 connection(s)
I20260926 14:53:37.471801 28995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:53:37.472012 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 28995
I20260926 14:53:37.472472 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:35281
--local_ip_for_outbound_sockets=127.12.68.193
--tserver_master_addrs=127.12.68.254:45237
--webserver_port=42411
--webserver_interface=127.12.68.193
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:53:37.476301 29063 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:53:37.477771 29063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Bootstrap starting.
I20260926 14:53:37.493533 29063 log.cc:824] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:37.503661 29063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:53:37.504004 29063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Bootstrap complete.
I20260926 14:53:37.507505 29063 raft_consensus.cc:348] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } }
I20260926 14:53:37.507849 29063 raft_consensus.cc:729] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b999bf78bccc4b56af25a62df1afc8c9, State: Initialized, Role: FOLLOWER
I20260926 14:53:37.508286 29063 consensus_queue.cc:260] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } }
I20260926 14:53:37.508391 29063 raft_consensus.cc:388] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:53:37.508448 29063 raft_consensus.cc:482] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:53:37.508535 29063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 2 FOLLOWER]: Advancing to term 3
I20260926 14:53:37.510149 29063 raft_consensus.cc:504] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } }
I20260926 14:53:37.510425 29063 leader_election.cc:302] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b999bf78bccc4b56af25a62df1afc8c9; no voters:
I20260926 14:53:37.510684 29063 leader_election.cc:288] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 14:53:37.510762 29067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 14:53:37.510936 29067 raft_consensus.cc:686] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [term 3 LEADER]: Becoming Leader. State: Replica: b999bf78bccc4b56af25a62df1afc8c9, State: Running, Role: LEADER
I20260926 14:53:37.511199 29067 consensus_queue.cc:237] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } }
I20260926 14:53:37.511780 29063 sys_catalog.cc:564] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:53:37.512079 29067 sys_catalog.cc:455] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b999bf78bccc4b56af25a62df1afc8c9. Latest consensus state: current_term: 3 leader_uuid: "b999bf78bccc4b56af25a62df1afc8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } } }
I20260926 14:53:37.512279 29067 sys_catalog.cc:458] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:37.513255 29071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:53:37.513387 29068 sys_catalog.cc:455] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b999bf78bccc4b56af25a62df1afc8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999bf78bccc4b56af25a62df1afc8c9" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45237 } } }
I20260926 14:53:37.513484 29068 sys_catalog.cc:458] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:37.517832 29071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:53:37.518437 29071 catalog_manager.cc:1259] Loaded cluster ID: c05b244356ea4c8daa63becf549718a3
I20260926 14:53:37.518494 29071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:53:37.520850 29071 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:53:37.521545 29071 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Loaded TSK: 0
I20260926 14:53:37.521621 29071 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b999bf78bccc4b56af25a62df1afc8c9: Loaded TSK: 1
F20260926 14:53:37.522295 29071 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbf9efe49ad google::LogMessage::Fail() at ??:0
@ 0x7fbf9efe788a google::LogMessage::SendToLog() at ??:0
@ 0x7fbf9efe44de google::LogMessage::Flush() at ??:0
@ 0x7fbf9efe5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbfa1ca818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbfa1ca0f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbfa99497bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbfa992399c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbfa9924178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fbfa9965df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fbfa998482b std::function<>::operator()() at ??:0
@ 0x7fbfa9923b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fbfa9924f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbfa9921579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fbfa9965640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbfa12363d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbfa1236c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbfa12383f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbfa1227512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbfa286b6db start_thread at ??:0
@ 0x7fbf9dae261f clone at ??:0
W20260926 14:53:37.596128 29065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:37.596411 29065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:37.596472 29065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:37.602276 29065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:37.602342 29065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:37.602423 29065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:53:37.609879 29065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:35281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=42411
--tserver_master_addrs=127.12.68.254:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:37.611071 29065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:37.612353 29065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:37.622617 29110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:37.622617 29113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:37.622685 29111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:37.623324 29065 server_base.cc:1034] running on GCE node
I20260926 14:53:37.623677 29065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:37.624270 29065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:37.625422 29065 hybrid_clock.cc:648] HybridClock initialized: now 1790434417625390 us; error 47 us; skew 500 ppm
Sep 26 14:53:37 dist-test-slave-24mn krb5kdc[27922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434417, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:37.627820 29065 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:53:37.628615 29065 webserver.cc:460] Webserver started at http://127.12.68.193:42411/ using document root <none> and password file <none>
I20260926 14:53:37.629101 29065 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:37.629155 29065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:37.632740 29065 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:53:37.634822 29120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:37.636097 29065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:53:37.636194 29065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "f69c45c6a4d946569810a68678df60a6"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "55cc0d15105a14b142b4bd6bd67dd96e"
server_key_iv: "d3e6c17ed933429f5917bbcc2c58ddea"
server_key_version: "encryptionkey@0"
I20260926 14:53:37.636675 29065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:37.663592 29065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:37.666488 29065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:37.666643 29065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:37.667169 29065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:37.668242 29127 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:53:37.673408 29065 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:53:37.673478 29065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 14:53:37.673540 29065 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:53:37.675227 29065 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:53:37.675269 29065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 14:53:37.675570 29127 tablet_bootstrap.cc:492] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6: Bootstrap starting.
I20260926 14:53:37.681780 29127 log.cc:824] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:37.684624 29065 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:35281
I20260926 14:53:37.684664 29235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:35281 every 8 connection(s)
I20260926 14:53:37.685487 29065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:53:37.688977 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 29065
I20260926 14:53:37.689424 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:42911
--local_ip_for_outbound_sockets=127.12.68.194
--tserver_master_addrs=127.12.68.254:45237
--webserver_port=35533
--webserver_interface=127.12.68.194
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:53:37.690915 29127 tablet_bootstrap.cc:492] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:53:37.691360 29127 tablet_bootstrap.cc:492] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6: Bootstrap complete.
I20260926 14:53:37.691706 29127 ts_tablet_manager.cc:1397] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20260926 14:53:37.695524 29127 raft_consensus.cc:348] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:37.695982 29127 raft_consensus.cc:729] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69c45c6a4d946569810a68678df60a6, State: Initialized, Role: FOLLOWER
I20260926 14:53:37.696448 29127 consensus_queue.cc:260] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:37.697234 29127 ts_tablet_manager.cc:1428] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
W20260926 14:53:37.753878 29238 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:53:37.687278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:34141)
0926 14:53:37.687704 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:37.687753 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:37.688955 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:37.753182 (+ 64227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
W20260926 14:53:37.754446 29236 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:45237 (0 consecutive failures): Network error: Failed to ping master at 127.12.68.254:45237: Client connection negotiation failed: client connection to 127.12.68.254:45237: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:53:37.755148 29238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:37.754911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:35307)
0926 14:53:37.755033 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:37.755094 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 14:53:37.755770 29238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:37.755592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:57117)
0926 14:53:37.755681 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:37.755722 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 14:53:37.755934 29236 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:53:37.812985 29239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:37.813201 29239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:37.813241 29239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:37.817085 29239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:37.817131 29239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:37.817200 29239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:53:37.822065 29239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:42911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=35533
--tserver_master_addrs=127.12.68.254:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:37.822911 29239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:37.823773 29239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:37.832365 29246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:37.832372 29248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:37.832402 29239 server_base.cc:1034] running on GCE node
W20260926 14:53:37.832378 29245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:37.832868 29239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:37.833417 29239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:37.834553 29239 hybrid_clock.cc:648] HybridClock initialized: now 1790434417834531 us; error 35 us; skew 500 ppm
Sep 26 14:53:37 dist-test-slave-24mn krb5kdc[27922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434417, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:37.836994 29239 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:53:37.837713 29239 webserver.cc:460] Webserver started at http://127.12.68.194:35533/ using document root <none> and password file <none>
I20260926 14:53:37.838200 29239 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:37.838265 29239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:37.841748 29239 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:53:37.843541 29255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:37.844712 29239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:53:37.844801 29239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "f1bed65d37794758b8a05c6bf9196d60"
format_stamp: "Formatted at 2026-09-26 14:52:55 on dist-test-slave-24mn"
server_key: "f628e21e37a9bcf4f2966b8162850f33"
server_key_iv: "0a47472a4f95d6d57ad9d0a0ed9fc6c0"
server_key_version: "encryptionkey@0"
I20260926 14:53:37.845276 29239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:37.862313 29239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:37.865038 29239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:37.865166 29239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:37.865648 29239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:37.866348 29239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:37.866389 29239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:37.866422 29239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:37.866441 29239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:37.874172 29239 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:42911
I20260926 14:53:37.874200 29368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:42911 every 8 connection(s)
I20260926 14:53:37.875022 29239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:53:37.875916 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 29239
I20260926 14:53:37.876331 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:33757
--local_ip_for_outbound_sockets=127.12.68.195
--tserver_master_addrs=127.12.68.254:45237
--webserver_port=40273
--webserver_interface=127.12.68.195
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:53:37.878973 29371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:37.877099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:39067)
0926 14:53:37.877561 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:37.878085 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":137,"threads_started":1}
W20260926 14:53:37.879460 29369 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:45237 (0 consecutive failures): Network error: Failed to ping master at 127.12.68.254:45237: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
I20260926 14:53:37.879992 29371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:37.879775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:60839)
0926 14:53:37.879870 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:37.879936 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 14:53:37.880548 29371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:37.880384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:37419)
0926 14:53:37.880467 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:37.880503 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:53:37.880695 29369 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:53:37.978240 29372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:37.978452 29372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:37.978492 29372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:37.982342 29372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:37.982386 29372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:37.982441 29372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:53:37.987341 29372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:33757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=40273
--tserver_master_addrs=127.12.68.254:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:37.988235 29372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:37.989112 29372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:37.997673 29377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:37.997773 29372 server_base.cc:1034] running on GCE node
W20260926 14:53:37.997677 29378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:37.997692 29380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:37.998260 29372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:37.998824 29372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:37.999956 29372 hybrid_clock.cc:648] HybridClock initialized: now 1790434417999936 us; error 31 us; skew 500 ppm
Sep 26 14:53:37 dist-test-slave-24mn krb5kdc[27922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434417, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:38.002454 29372 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:53:38.003178 29372 webserver.cc:460] Webserver started at http://127.12.68.195:40273/ using document root <none> and password file <none>
I20260926 14:53:38.003651 29372 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:38.003710 29372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:38.007182 29372 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:53:38.009039 29387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:38.010113 29372 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:53:38.010218 29372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "259cbbb3436140beb73e9ac00a6eb9e6"
format_stamp: "Formatted at 2026-09-26 14:52:56 on dist-test-slave-24mn"
server_key: "d6386eb578f33ac3f363874c7865ab62"
server_key_iv: "0368d0b053cf65ce5fe8d6b472db736a"
server_key_version: "encryptionkey@0"
I20260926 14:53:38.010711 29372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:38.029016 29372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:38.031623 29372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:38.031778 29372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:38.032320 29372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:38.033046 29372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:38.033090 29372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:38.033138 29372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:38.033166 29372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:38.040971 29372 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:33757
I20260926 14:53:38.041049 29500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:33757 every 8 connection(s)
I20260926 14:53:38.041920 29372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:53:38.042354 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 29372
I20260926 14:53:38.045393 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:38.043943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:59243)
0926 14:53:38.044357 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:38.044746 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":90,"threads_started":1}
W20260926 14:53:38.045830 29501 heartbeater.cc:644] Failed to heartbeat to 127.12.68.254:45237 (0 consecutive failures): Network error: Failed to ping master at 127.12.68.254:45237: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
I20260926 14:53:38.046300 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:38.046116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:34561)
0926 14:53:38.046207 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:38.046250 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 14:53:38.047355 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:38.047142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:43955)
0926 14:53:38.047249 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:38.047298 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 14:53:38.047708 29501 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:53:38.756793 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:38.756421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:60383)
0926 14:53:38.756644 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:38.756700 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
I20260926 14:53:38.881537 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:38.881183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:40199)
0926 14:53:38.881394 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:38.881444 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":61,"threads_started":1}
I20260926 14:53:39.048492 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:39.048151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:60265)
0926 14:53:39.048358 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:39.048407 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
I20260926 14:53:39.132292 29508 raft_consensus.cc:482] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:53:39.132479 29508 raft_consensus.cc:504] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:39.133512 29508 leader_election.cc:288] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911), 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757)
Sep 26 14:53:39 dist-test-slave-24mn krb5kdc[27922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434417, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.194@KRBTEST.COM
Sep 26 14:53:39 dist-test-slave-24mn krb5kdc[27922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434417, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.195@KRBTEST.COM
I20260926 14:53:39.146001 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:39.133919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:33757 (local address 127.12.68.193:51289)
0926 14:53:39.134182 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:39.134193 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:39.134287 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:39.135629 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:39.135636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:39.136198 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:39.136714 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:39.136728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:39.137927 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:39.137929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:39.138806 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:39.138815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:39.139483 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:39.139486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:39.139681 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:39.140393 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:39.140405 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:39.142133 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:39.143867 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:39.143872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:39.143910 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:39.144157 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:39.144456 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:39.144459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:39.144461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:39.144578 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:39.145123 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:39.145129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:39.145353 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 14:53:39.145496 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 14:53:39.145810 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":137,"spinlock_wait_cycles":46848,"thread_start_us":46,"threads_started":1}
I20260926 14:53:39.146018 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:39.134093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:51289 (local address 127.12.68.195:33757)
0926 14:53:39.134402 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:39.134420 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:39.134451 (+ 31us) server_negotiation.cc:407] Connection header received
0926 14:53:39.135043 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:39.135048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:39.135330 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:39.135591 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:39.136870 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:39.137791 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:39.138916 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:39.139376 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:39.139523 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:39.142293 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:39.142297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:39.142307 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:39.142334 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:39.143755 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:39.144255 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:39.144260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:39.144264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:39.144373 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:39.144669 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:39.144672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:39.144674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:39.145018 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:39.145127 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:39.145632 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 14:53:39.145870 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
I20260926 14:53:39.146046 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:39.133871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:40143 (local address 127.12.68.194:42911)
0926 14:53:39.134158 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:39.134188 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:39.134235 (+ 47us) server_negotiation.cc:407] Connection header received
0926 14:53:39.134932 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:39.134938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:39.135274 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:39.135533 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:39.136864 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:39.137793 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:39.138902 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:39.139376 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:39.139516 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:39.142062 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:39.142069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:39.142083 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:39.142113 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:39.143938 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:39.144332 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:39.144338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:39.144343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:39.144446 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:39.144680 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:39.144683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:39.144685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:39.145018 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:39.145127 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:39.145628 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 14:53:39.145868 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260926 14:53:39.146046 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:39.133673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:42911 (local address 127.12.68.193:40143)
0926 14:53:39.133796 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:39.133810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:39.133916 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:39.135630 (+ 1714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:39.135636 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:39.136197 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:39.136713 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:39.136727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:39.137931 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:39.137934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:39.138798 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:39.138806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:39.139482 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:39.139485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:39.139684 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:39.140326 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:39.140342 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:39.141841 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:39.144044 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:39.144047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:39.144049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:39.144235 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:39.144530 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:39.144533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:39.144534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:39.144605 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:39.145123 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:39.145129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:39.145354 (+ 225us) client_negotiation.cc:769] Sending connection context
0926 14:53:39.145497 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 14:53:39.145886 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":302,"spinlock_wait_cycles":14720}
W20260926 14:53:39.148641 29123 leader_election.cc:341] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911): Not found: Tablet not found: de203d0df61842f68d79eb3c11511c7d
W20260926 14:53:39.148824 29123 leader_election.cc:341] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757): Not found: Tablet not found: de203d0df61842f68d79eb3c11511c7d
I20260926 14:53:39.148878 29123 leader_election.cc:302] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f69c45c6a4d946569810a68678df60a6; no voters: 259cbbb3436140beb73e9ac00a6eb9e6, f1bed65d37794758b8a05c6bf9196d60
I20260926 14:53:39.149111 29508 raft_consensus.cc:2726] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:53:39.757853 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:39.757463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:48841)
0926 14:53:39.757700 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:39.757754 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":70,"threads_started":1}
I20260926 14:53:39.882462 29513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:39.882134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:34987)
0926 14:53:39.882335 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:39.882384 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":64,"threads_started":1}
I20260926 14:53:40.049398 29514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:40.049030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:43231)
0926 14:53:40.049269 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:40.049318 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":64,"threads_started":1}
I20260926 14:53:40.758894 29515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:40.758518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:46923)
0926 14:53:40.758745 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:40.758803 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260926 14:53:40.883443 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:40.883058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:59975)
0926 14:53:40.883295 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:40.883347 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
I20260926 14:53:41.050367 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:41.050002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:55491)
0926 14:53:41.050213 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:41.050271 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":67,"threads_started":1}
I20260926 14:53:41.228518 29518 raft_consensus.cc:482] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:53:41.228643 29518 raft_consensus.cc:504] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:41.229031 29518 leader_election.cc:288] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911), 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757)
W20260926 14:53:41.229851 29123 leader_election.cc:341] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757): Not found: Tablet not found: de203d0df61842f68d79eb3c11511c7d
W20260926 14:53:41.229983 29123 leader_election.cc:341] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911): Not found: Tablet not found: de203d0df61842f68d79eb3c11511c7d
I20260926 14:53:41.230037 29123 leader_election.cc:302] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f69c45c6a4d946569810a68678df60a6; no voters: 259cbbb3436140beb73e9ac00a6eb9e6, f1bed65d37794758b8a05c6bf9196d60
I20260926 14:53:41.230187 29518 raft_consensus.cc:2726] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:53:41.759971 29519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:41.759565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:56649)
0926 14:53:41.759793 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:41.759843 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260926 14:53:41.884473 29520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:41.884112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:39893)
0926 14:53:41.884340 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:41.884390 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":76,"threads_started":1}
I20260926 14:53:42.051383 29521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:42.051004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:60839)
0926 14:53:42.051254 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:42.051306 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":83,"threads_started":1}
I20260926 14:53:42.761142 29522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:42.760713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:53959)
0926 14:53:42.760998 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:42.761048 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":74,"threads_started":1}
I20260926 14:53:42.885504 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:42.885136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:44533)
0926 14:53:42.885365 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:42.885416 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
I20260926 14:53:43.052456 29524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:43.052087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:48851)
0926 14:53:43.052326 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:43.052376 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
I20260926 14:53:43.349284 29525 raft_consensus.cc:482] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:53:43.349423 29525 raft_consensus.cc:504] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:43.349787 29525 leader_election.cc:288] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911), 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757)
W20260926 14:53:43.350462 29123 leader_election.cc:341] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911): Not found: Tablet not found: de203d0df61842f68d79eb3c11511c7d
W20260926 14:53:43.350574 29123 leader_election.cc:341] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757): Not found: Tablet not found: de203d0df61842f68d79eb3c11511c7d
I20260926 14:53:43.350608 29123 leader_election.cc:302] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f69c45c6a4d946569810a68678df60a6; no voters: 259cbbb3436140beb73e9ac00a6eb9e6, f1bed65d37794758b8a05c6bf9196d60
I20260926 14:53:43.350750 29525 raft_consensus.cc:2726] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:53:43.762135 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:43.761770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:50125)
0926 14:53:43.761995 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:43.762043 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260926 14:53:43.886583 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:43.886191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:59505)
0926 14:53:43.886437 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:43.886493 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20260926 14:53:44.053408 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:44.053032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:49591)
0926 14:53:44.053260 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:44.053311 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260926 14:53:44.763175 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:44.762781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:49397)
0926 14:53:44.763011 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:44.763082 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20260926 14:53:44.887633 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:44.887241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:43499)
0926 14:53:44.887500 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:44.887549 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":67,"threads_started":1}
I20260926 14:53:45.054471 29531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:45.054018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:58871)
0926 14:53:45.054322 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:45.054374 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":96,"threads_started":1}
I20260926 14:53:45.764250 29532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:45.763825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:56249)
0926 14:53:45.764096 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:45.764157 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":99,"threads_started":1}
I20260926 14:53:45.888648 29533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:45.888245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:34439)
0926 14:53:45.888499 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:45.888558 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20260926 14:53:46.055496 29534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:46.055117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:56885)
0926 14:53:46.055348 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:46.055401 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260926 14:53:46.765295 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:46.764922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:56687)
0926 14:53:46.765154 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:46.765203 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
I20260926 14:53:46.889670 29536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:46.889296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:41451)
0926 14:53:46.889533 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:46.889582 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
I20260926 14:53:47.056552 29537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:47.056180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:56875)
0926 14:53:47.056408 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:47.056459 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
I20260926 14:53:47.766415 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:47.766028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:42797)
0926 14:53:47.766274 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:47.766323 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260926 14:53:47.890714 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:47.890322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:59309)
0926 14:53:47.890576 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:47.890625 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260926 14:53:48.044337 29540 raft_consensus.cc:482] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:53:48.044448 29540 raft_consensus.cc:504] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:48.044823 29540 leader_election.cc:288] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911), 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757)
W20260926 14:53:48.045542 29123 leader_election.cc:341] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911): Not found: Tablet not found: de203d0df61842f68d79eb3c11511c7d
W20260926 14:53:48.045680 29123 leader_election.cc:341] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757): Not found: Tablet not found: de203d0df61842f68d79eb3c11511c7d
I20260926 14:53:48.045732 29123 leader_election.cc:302] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f69c45c6a4d946569810a68678df60a6; no voters: 259cbbb3436140beb73e9ac00a6eb9e6, f1bed65d37794758b8a05c6bf9196d60
I20260926 14:53:48.045882 29540 raft_consensus.cc:2726] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:53:48.057507 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:48.057154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:35025)
0926 14:53:48.057371 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:48.057422 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":59,"threads_started":1}
I20260926 14:53:48.767526 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:48.767114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:51737)
0926 14:53:48.767378 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:48.767433 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":72,"threads_started":1}
I20260926 14:53:48.891748 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:48.891364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:53925)
0926 14:53:48.891603 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:48.891657 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260926 14:53:49.058565 29544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:49.058162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:41583)
0926 14:53:49.058419 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:49.058476 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":71,"threads_started":1}
I20260926 14:53:49.768671 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:49.768286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:40003)
0926 14:53:49.768523 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:49.768576 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260926 14:53:49.892838 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:49.892432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:39237)
0926 14:53:49.892687 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:49.892741 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260926 14:53:50.059571 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:50.059191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:60077)
0926 14:53:50.059428 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:50.059480 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260926 14:53:50.117040 29548 raft_consensus.cc:482] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:53:50.117167 29548 raft_consensus.cc:504] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1bed65d37794758b8a05c6bf9196d60" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 42911 } } peers { permanent_uuid: "259cbbb3436140beb73e9ac00a6eb9e6" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 33757 } } peers { permanent_uuid: "f69c45c6a4d946569810a68678df60a6" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 35281 } }
I20260926 14:53:50.117555 29548 leader_election.cc:288] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911), 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757)
W20260926 14:53:50.118314 29123 leader_election.cc:341] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1bed65d37794758b8a05c6bf9196d60 (127.12.68.194:42911): Not found: Tablet not found: de203d0df61842f68d79eb3c11511c7d
W20260926 14:53:50.118456 29123 leader_election.cc:341] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 259cbbb3436140beb73e9ac00a6eb9e6 (127.12.68.195:33757): Not found: Tablet not found: de203d0df61842f68d79eb3c11511c7d
I20260926 14:53:50.118507 29123 leader_election.cc:302] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f69c45c6a4d946569810a68678df60a6; no voters: 259cbbb3436140beb73e9ac00a6eb9e6, f1bed65d37794758b8a05c6bf9196d60
I20260926 14:53:50.118657 29548 raft_consensus.cc:2726] T de203d0df61842f68d79eb3c11511c7d P f69c45c6a4d946569810a68678df60a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:53:50.769708 29549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:50.769324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:49347)
0926 14:53:50.769560 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:50.769614 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260926 14:53:50.893872 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:50.893510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:53153)
0926 14:53:50.893738 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:50.893786 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
I20260926 14:53:51.060693 29551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:51.060273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:49817)
0926 14:53:51.060546 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:51.060602 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
I20260926 14:53:51.771204 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:51.770395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:43515)
0926 14:53:51.770678 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:51.770737 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":71,"threads_started":1}
I20260926 14:53:51.894989 29553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:51.894568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:57091)
0926 14:53:51.894806 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:51.894882 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260926 14:53:52.061741 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:52.061367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.195:47281)
0926 14:53:52.061586 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:52.061640 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260926 14:53:52.772488 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:52.772125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.193:56109)
0926 14:53:52.772346 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:52.772397 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260926 14:53:52.896196 29556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:52.895779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45237 (local address 127.12.68.194:57775)
0926 14:53:52.896052 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:52.896103 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.254:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":71,"threads_started":1}
I20260926 14:53:53.049580 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 29065
I20260926 14:53:53.054620 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 29239
I20260926 14:53:53.059221 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 29372
2026-09-26T14:53:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58705 ms)
[----------] 30 tests from SecurityITest (228629 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:53:53 dist-test-slave-24mn krb5kdc[29559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:53:53 dist-test-slave-24mn krb5kdc[29559](info): set up 2 sockets
Sep 26 14:53:53 dist-test-slave-24mn krb5kdc[29559](info): commencing operation
krb5kdc: starting...
W20260926 14:53:54.011437 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:53:54 dist-test-slave-24mn krb5kdc[29559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:53:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:53:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:53:54.122733 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:37355
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:37355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:54.226341 29572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:54.226586 29572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:54.226626 29572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:54.230562 29572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:53:54.230612 29572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:54.230630 29572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:54.230645 29572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:53:54.230659 29572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:53:54.235739 29572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:37355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:37355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:54.236701 29572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:54.237696 29572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:54.245522 29578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:54.245522 29580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:54.245596 29572 server_base.cc:1034] running on GCE node
W20260926 14:53:54.245546 29577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:54.246248 29572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:54.247251 29572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:54.248399 29572 hybrid_clock.cc:648] HybridClock initialized: now 1790434434248386 us; error 25 us; skew 500 ppm
Sep 26 14:53:54 dist-test-slave-24mn krb5kdc[29559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:54.250768 29572 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:53:54.251551 29572 webserver.cc:460] Webserver started at http://127.12.68.254:44855/ using document root <none> and password file <none>
I20260926 14:53:54.252082 29572 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:54.252146 29572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:54.252332 29572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:54.254194 29572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "bf8e7fcb36c2401a9711b3aa7b068c5c"
format_stamp: "Formatted at 2026-09-26 14:53:54 on dist-test-slave-24mn"
server_key: "124ffd15ae4087dec1139b0fe44e3761"
server_key_iv: "b1f396946460bf1bbb0211efb9bafea7"
server_key_version: "encryptionkey@0"
I20260926 14:53:54.254688 29572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "bf8e7fcb36c2401a9711b3aa7b068c5c"
format_stamp: "Formatted at 2026-09-26 14:53:54 on dist-test-slave-24mn"
server_key: "124ffd15ae4087dec1139b0fe44e3761"
server_key_iv: "b1f396946460bf1bbb0211efb9bafea7"
server_key_version: "encryptionkey@0"
I20260926 14:53:54.258107 29572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:53:54.260339 29587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:54.261559 29572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:53:54.261664 29572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "bf8e7fcb36c2401a9711b3aa7b068c5c"
format_stamp: "Formatted at 2026-09-26 14:53:54 on dist-test-slave-24mn"
server_key: "124ffd15ae4087dec1139b0fe44e3761"
server_key_iv: "b1f396946460bf1bbb0211efb9bafea7"
server_key_version: "encryptionkey@0"
I20260926 14:53:54.261761 29572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:54.285810 29572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:54.288599 29572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:54.288769 29572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:54.295303 29572 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:37355
I20260926 14:53:54.295315 29639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:37355 every 8 connection(s)
I20260926 14:53:54.296212 29572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:53:54.297921 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 29572
I20260926 14:53:54.298032 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:53:54.298288 12563 external_mini_cluster.cc:1428] Setting key 3865d73f846aadf4eb39b125ce641d4b
I20260926 14:53:54.298979 29640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:54.305383 29640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c: Bootstrap starting.
Sep 26 14:53:54 dist-test-slave-24mn krb5kdc[29559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:53:54.307512 29640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:54.308362 29640 log.cc:824] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:54.310408 29640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c: No bootstrap required, opened a new log
I20260926 14:53:54.312368 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:54.299492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.12.68.254:37355)
0926 14:53:54.299900 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:54.299919 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:54.299960 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:53:54.300765 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:54.300787 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:54.301178 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:54.301529 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:54.302142 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.303585 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.304468 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.305121 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.305274 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:54.307577 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:54.307583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:54.307599 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:54.307630 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:54.309354 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:54.309739 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:54.309744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:54.309750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:54.309875 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:54.310096 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:54.310099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:54.310100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:54.310573 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:54.310753 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:54.310945 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 14:53:54.311220 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260926 14:53:54.313995 29640 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8e7fcb36c2401a9711b3aa7b068c5c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37355 } }
I20260926 14:53:54.314206 29640 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:54.314271 29640 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8e7fcb36c2401a9711b3aa7b068c5c, State: Initialized, Role: FOLLOWER
I20260926 14:53:54.314690 29640 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8e7fcb36c2401a9711b3aa7b068c5c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37355 } }
I20260926 14:53:54.314810 29640 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:53:54.314867 29640 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:53:54.314955 29640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:54.315703 29640 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8e7fcb36c2401a9711b3aa7b068c5c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37355 } }
I20260926 14:53:54.316026 29640 leader_election.cc:302] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf8e7fcb36c2401a9711b3aa7b068c5c; no voters:
I20260926 14:53:54.316287 29640 leader_election.cc:288] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:53:54.316371 29645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:53:54.316586 29645 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [term 1 LEADER]: Becoming Leader. State: Replica: bf8e7fcb36c2401a9711b3aa7b068c5c, State: Running, Role: LEADER
I20260926 14:53:54.316881 29645 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8e7fcb36c2401a9711b3aa7b068c5c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37355 } }
I20260926 14:53:54.317327 29640 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:53:54.318267 29646 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf8e7fcb36c2401a9711b3aa7b068c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8e7fcb36c2401a9711b3aa7b068c5c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37355 } } }
I20260926 14:53:54.318378 29646 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:54.318480 29647 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf8e7fcb36c2401a9711b3aa7b068c5c. Latest consensus state: current_term: 1 leader_uuid: "bf8e7fcb36c2401a9711b3aa7b068c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8e7fcb36c2401a9711b3aa7b068c5c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37355 } } }
I20260926 14:53:54.318558 29647 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:54.318920 29654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:53:54.321928 29654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:53:54.326385 29654 catalog_manager.cc:1347] Generated new cluster ID: f6430be2a04341bc805e7f46669b67d8
I20260926 14:53:54.326442 29654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:53:54.336158 29654 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:53:54.336944 29654 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:53:54.346864 29654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c: Generated new TSK 0
I20260926 14:53:54.347540 29654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:53:54.379505 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:37355
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:54.483451 29666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:54.483682 29666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:54.483739 29666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:54.487605 29666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:54.487656 29666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:54.487735 29666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:53:54.492754 29666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:37355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:54.493649 29666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:54.494560 29666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:54.503351 29672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:54.503362 29671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:54.503412 29674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:54.503450 29666 server_base.cc:1034] running on GCE node
I20260926 14:53:54.504035 29666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:54.504616 29666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:54.505756 29666 hybrid_clock.cc:648] HybridClock initialized: now 1790434434505715 us; error 58 us; skew 500 ppm
Sep 26 14:53:54 dist-test-slave-24mn krb5kdc[29559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:54.508205 29666 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:53:54.508934 29666 webserver.cc:460] Webserver started at http://127.12.68.193:45891/ using document root <none> and password file <none>
I20260926 14:53:54.509420 29666 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:54.509481 29666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:54.509650 29666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:54.511505 29666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "496dbffab5e24c8d9856bf3d82ede24a"
format_stamp: "Formatted at 2026-09-26 14:53:54 on dist-test-slave-24mn"
server_key: "23a05bf98e9df5f46f749490e5ce99b3"
server_key_iv: "ff3479d51d37b16866257668a9fe0b7e"
server_key_version: "encryptionkey@0"
I20260926 14:53:54.512027 29666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "496dbffab5e24c8d9856bf3d82ede24a"
format_stamp: "Formatted at 2026-09-26 14:53:54 on dist-test-slave-24mn"
server_key: "23a05bf98e9df5f46f749490e5ce99b3"
server_key_iv: "ff3479d51d37b16866257668a9fe0b7e"
server_key_version: "encryptionkey@0"
I20260926 14:53:54.515307 29666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:53:54.517509 29681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:54.518679 29666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:53:54.518783 29666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "496dbffab5e24c8d9856bf3d82ede24a"
format_stamp: "Formatted at 2026-09-26 14:53:54 on dist-test-slave-24mn"
server_key: "23a05bf98e9df5f46f749490e5ce99b3"
server_key_iv: "ff3479d51d37b16866257668a9fe0b7e"
server_key_version: "encryptionkey@0"
I20260926 14:53:54.518865 29666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:54.529603 29666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:54.532217 29666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:54.532367 29666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:54.532859 29666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:54.533582 29666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:54.533627 29666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:54.533674 29666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:54.533701 29666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:54.541945 29666 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:40545
I20260926 14:53:54.541980 29794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:40545 every 8 connection(s)
I20260926 14:53:54.542730 29666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:53:54.544866 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 29666
I20260926 14:53:54.544982 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:53:54.545295 12563 external_mini_cluster.cc:1428] Setting key 098a71d3a4b7dfde455ebebacfe4b399
Sep 26 14:53:54 dist-test-slave-24mn krb5kdc[29559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
I20260926 14:53:54.556267 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:54.544398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:47441 (local address 127.12.68.254:37355)
0926 14:53:54.544518 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:54.544522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:54.545373 (+ 851us) server_negotiation.cc:407] Connection header received
0926 14:53:54.546872 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:54.546874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:54.546916 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:54.546972 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:54.548392 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.549200 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.550288 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.550656 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.550761 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:54.553111 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:54.553114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:54.553116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:54.553139 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:54.554390 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:54.554813 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:54.554815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:54.554816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:54.554883 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:54.555231 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:54.555233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:54.555234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:54.555392 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:54.555462 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:54.556045 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 14:53:54.556148 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
I20260926 14:53:54.557150 29797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:54.544751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:37355 (local address 127.12.68.193:47441)
0926 14:53:54.545226 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:54.545278 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:54.546643 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:54.547171 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:54.547179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:54.547594 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:54.548243 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:54.548263 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.549312 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.549315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:54.550170 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:54.550179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.550786 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.550789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:54.550936 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:54.551440 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:54.551455 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:54.552950 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:54.554486 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:54.554490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:54.554508 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:54.554734 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:54.555018 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:54.555021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:54.555023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:54.555158 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:54.555505 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:54.555510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:54.555782 (+ 272us) client_negotiation.cc:769] Sending connection context
0926 14:53:54.556000 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 14:53:54.556249 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
I20260926 14:53:54.558284 29795 heartbeater.cc:347] Connected to a master server at 127.12.68.254:37355
I20260926 14:53:54.558499 29795 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:54.558935 29795 heartbeater.cc:511] Master 127.12.68.254:37355 requested a full tablet report, sending...
I20260926 14:53:54.560374 29604 ts_manager.cc:194] Registered new tserver with Master: 496dbffab5e24c8d9856bf3d82ede24a (127.12.68.193:40545)
I20260926 14:53:54.561431 29604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:47441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:53:54.570276 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:37355
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:54.673342 29800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:54.673583 29800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:54.673640 29800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:54.677599 29800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:54.677651 29800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:54.677727 29800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:53:54.682868 29800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:37355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:54.683835 29800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:54.684804 29800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:54.693753 29808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:54.693781 29806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:54.693804 29805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:54.693969 29800 server_base.cc:1034] running on GCE node
I20260926 14:53:54.694352 29800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:54.694962 29800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:54.696112 29800 hybrid_clock.cc:648] HybridClock initialized: now 1790434434696084 us; error 38 us; skew 500 ppm
Sep 26 14:53:54 dist-test-slave-24mn krb5kdc[29559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:54.704922 29800 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:53:54.705751 29800 webserver.cc:460] Webserver started at http://127.12.68.194:44075/ using document root <none> and password file <none>
I20260926 14:53:54.706252 29800 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:54.706313 29800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:54.706485 29800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:54.708437 29800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "6fd3cc95fa774471a7d61eea86ed6e0f"
format_stamp: "Formatted at 2026-09-26 14:53:54 on dist-test-slave-24mn"
server_key: "c6b932b51c6517bd979ae53d81359b8a"
server_key_iv: "681b7a45dd0ea8e38a1cf6f18c52f5e0"
server_key_version: "encryptionkey@0"
I20260926 14:53:54.708946 29800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "6fd3cc95fa774471a7d61eea86ed6e0f"
format_stamp: "Formatted at 2026-09-26 14:53:54 on dist-test-slave-24mn"
server_key: "c6b932b51c6517bd979ae53d81359b8a"
server_key_iv: "681b7a45dd0ea8e38a1cf6f18c52f5e0"
server_key_version: "encryptionkey@0"
I20260926 14:53:54.712316 29800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:53:54.714498 29815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:54.715600 29800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:53:54.715693 29800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "6fd3cc95fa774471a7d61eea86ed6e0f"
format_stamp: "Formatted at 2026-09-26 14:53:54 on dist-test-slave-24mn"
server_key: "c6b932b51c6517bd979ae53d81359b8a"
server_key_iv: "681b7a45dd0ea8e38a1cf6f18c52f5e0"
server_key_version: "encryptionkey@0"
I20260926 14:53:54.715773 29800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:54.727276 29800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:54.730803 29800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:54.730993 29800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:54.731653 29800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:54.732532 29800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:54.732589 29800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:54.732638 29800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:54.732657 29800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:54.742105 29800 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:38065
I20260926 14:53:54.742172 29928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:38065 every 8 connection(s)
I20260926 14:53:54.742858 29800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:53:54.745857 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 29800
I20260926 14:53:54.745987 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:53:54.746340 12563 external_mini_cluster.cc:1428] Setting key ec93189f364f3d97bdb0cf17ab1fb1a0
Sep 26 14:53:54 dist-test-slave-24mn krb5kdc[29559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:53:54.758484 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:54.745942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:58637 (local address 127.12.68.254:37355)
0926 14:53:54.746070 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:54.746075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:54.746093 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:53:54.748406 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:54.748410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:54.748462 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:54.748529 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:54.749675 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.750185 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.751293 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.751516 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.751607 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:54.754415 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:54.754419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:54.754422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:54.754448 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:54.756265 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:54.756841 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:54.756844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:54.756847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:54.756935 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:54.757307 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:54.757310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:54.757312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:54.757522 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:54.757617 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:54.758201 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 14:53:54.758319 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
I20260926 14:53:54.759311 29931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:54.744947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:37355 (local address 127.12.68.194:58637)
0926 14:53:54.745458 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:54.745514 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:54.748202 (+ 2688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:54.748695 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:54.748702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:54.749063 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:54.749546 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:54.749558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.750292 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.750295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:54.751189 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:54.751198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.751642 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.751646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:54.751820 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:54.752449 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:54.752469 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:54.754258 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:54.756396 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:54.756402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:54.756416 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:54.756734 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:54.757032 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:54.757035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:54.757037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:54.757218 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:54.757631 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:54.757637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:54.757927 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 14:53:54.758150 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 14:53:54.758459 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":131,"threads_started":1}
I20260926 14:53:54.760589 29929 heartbeater.cc:347] Connected to a master server at 127.12.68.254:37355
I20260926 14:53:54.760823 29929 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
I20260926 14:53:54.761387 29929 heartbeater.cc:511] Master 127.12.68.254:37355 requested a full tablet report, sending...
I20260926 14:53:54.762473 29604 ts_manager.cc:194] Registered new tserver with Master: 6fd3cc95fa774471a7d61eea86ed6e0f (127.12.68.194:38065)
I20260926 14:53:54.763221 29604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:58637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:53:54.775400 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:37355
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:54.881065 29934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:54.881294 29934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:54.881348 29934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:54.885246 29934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:54.885294 29934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:54.885370 29934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:53:54.890398 29934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:37355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:54.891307 29934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:54.892234 29934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:54.900851 29942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:54.900946 29934 server_base.cc:1034] running on GCE node
W20260926 14:53:54.900851 29939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:54.900861 29940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:54.901413 29934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:54.901971 29934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:54.903144 29934 hybrid_clock.cc:648] HybridClock initialized: now 1790434434903127 us; error 28 us; skew 500 ppm
Sep 26 14:53:54 dist-test-slave-24mn krb5kdc[29559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:54.905622 29934 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:53:54.906373 29934 webserver.cc:460] Webserver started at http://127.12.68.195:36909/ using document root <none> and password file <none>
I20260926 14:53:54.906848 29934 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:54.906904 29934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:54.907110 29934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:54.908985 29934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "228f789acd5f4488bf9c8c6138daac72"
format_stamp: "Formatted at 2026-09-26 14:53:54 on dist-test-slave-24mn"
server_key: "9dffe5f7865057ce4c977d6525ba0436"
server_key_iv: "3547d806a470e8a2a1dfdee2789123c0"
server_key_version: "encryptionkey@0"
I20260926 14:53:54.909488 29934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "228f789acd5f4488bf9c8c6138daac72"
format_stamp: "Formatted at 2026-09-26 14:53:54 on dist-test-slave-24mn"
server_key: "9dffe5f7865057ce4c977d6525ba0436"
server_key_iv: "3547d806a470e8a2a1dfdee2789123c0"
server_key_version: "encryptionkey@0"
I20260926 14:53:54.912858 29934 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:53:54.915067 29949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:54.916209 29934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:53:54.916313 29934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "228f789acd5f4488bf9c8c6138daac72"
format_stamp: "Formatted at 2026-09-26 14:53:54 on dist-test-slave-24mn"
server_key: "9dffe5f7865057ce4c977d6525ba0436"
server_key_iv: "3547d806a470e8a2a1dfdee2789123c0"
server_key_version: "encryptionkey@0"
I20260926 14:53:54.916397 29934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:54.934253 29934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:54.936771 29934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:54.936921 29934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:54.937422 29934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:54.938155 29934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:54.938197 29934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:54.938246 29934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:54.938273 29934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:54.945916 29934 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:44997
I20260926 14:53:54.945968 30062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:44997 every 8 connection(s)
I20260926 14:53:54.946724 29934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:53:54.950302 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 29934
I20260926 14:53:54.950418 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:53:54.950671 12563 external_mini_cluster.cc:1428] Setting key b7d5cfddac7a7de466bd574f0f902e1c
Sep 26 14:53:54 dist-test-slave-24mn krb5kdc[29559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:53:54.958729 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:54.948297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:58203 (local address 127.12.68.254:37355)
0926 14:53:54.948406 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:54.948410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:54.949127 (+ 717us) server_negotiation.cc:407] Connection header received
0926 14:53:54.950412 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:54.950416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:54.950476 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:54.950548 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:54.951833 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.952422 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.953504 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.953717 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.953782 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:54.955939 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:54.955942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:54.955944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:54.955966 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:54.957024 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:54.957447 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:54.957449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:54.957450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:54.957511 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:54.957755 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:54.957757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:54.957758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:54.957925 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:54.958004 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:54.958513 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 14:53:54.958617 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:53:54.959401 30065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:54.948607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:37355 (local address 127.12.68.195:58203)
0926 14:53:54.948995 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:54.949033 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:54.950219 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:54.950735 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:54.950744 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:54.951176 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:54.951709 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:54.951729 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.952519 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.952522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:54.953401 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:54.953411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.953814 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.953817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:54.953948 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:54.954441 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:54.954457 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:54.955766 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:54.957132 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:54.957137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:54.957146 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:54.957373 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:54.957567 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:54.957569 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:54.957578 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:54.957688 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:54.958079 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:54.958084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:54.958294 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 14:53:54.958453 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 14:53:54.958676 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":106,"threads_started":1}
I20260926 14:53:54.960444 30063 heartbeater.cc:347] Connected to a master server at 127.12.68.254:37355
I20260926 14:53:54.960654 30063 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:54.961086 30063 heartbeater.cc:511] Master 127.12.68.254:37355 requested a full tablet report, sending...
I20260926 14:53:54.962004 29604 ts_manager.cc:194] Registered new tserver with Master: 228f789acd5f4488bf9c8c6138daac72 (127.12.68.195:44997)
I20260926 14:53:54.962687 29604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:58203
I20260926 14:53:54.963989 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:53:54.972321 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:54.965891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.12.68.254:37355)
0926 14:53:54.966003 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:54.966007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:54.966188 (+ 181us) server_negotiation.cc:407] Connection header received
0926 14:53:54.966285 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:54.966288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:54.966327 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:54.966387 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:54.967000 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.967493 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.968323 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.968541 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.968620 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:54.969815 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:54.969819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:54.969821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:54.969844 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:54.970833 (+ 989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:54.971205 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:54.971208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:54.971210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:54.971279 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:54.971502 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:54.971505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:54.971507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:54.971699 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:54.971811 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:54.972067 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 14:53:54.972186 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:53:54.974794 29604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:53:54.977619 29604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:53:54.989835 30078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:54.985313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.12.68.194:38065)
0926 14:53:54.985787 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:54.985796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:54.985814 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:53:54.985880 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:54.985886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:54.986078 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:54.986212 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:54.986482 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.987328 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.988677 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.989241 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.989335 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:54.989467 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:54.989617 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 14:53:54.989702 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":130,"threads_started":1}
I20260926 14:53:54.989832 30076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:54.984886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:38065 (local address 127.0.0.1:48792)
0926 14:53:54.985111 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:54.985125 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:54.985208 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:54.986212 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:54.986218 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:54.986239 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:54.986363 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:54.986370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.987431 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.987434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:54.988556 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:54.988566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.989347 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.989350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:54.989450 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:54.989484 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 14:53:54.989578 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 14:53:54.989698 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":33,"spinlock_wait_cycles":768,"thread_start_us":87,"threads_started":1}
I20260926 14:53:54.991065 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:54.985241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:44997 (local address 127.0.0.1:48534)
0926 14:53:54.985563 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:54.985575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:54.985652 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:54.986225 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:54.986228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:54.986254 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:54.986371 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:54.986377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.988225 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.988229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:54.989610 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:54.989620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.990675 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.990678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:54.990761 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:54.990785 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:53:54.990860 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 14:53:54.990939 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":120,"spinlock_wait_cycles":2304,"thread_start_us":86,"threads_started":1}
I20260926 14:53:54.991128 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:54.985393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.12.68.195:44997)
0926 14:53:54.985835 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:54.985846 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:54.985867 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:53:54.985934 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:54.985939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:54.986108 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:54.986232 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:54.986919 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.988084 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.989748 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.990554 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.990654 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:54.990773 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:54.990909 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 14:53:54.990998 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":127,"threads_started":1}
I20260926 14:53:54.991225 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:54.984616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:40545 (local address 127.0.0.1:48736)
0926 14:53:54.984886 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:54.984927 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:54.985032 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:54.986801 (+ 1769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:54.986804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:54.986822 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:54.986913 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:54.986918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.988276 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.988279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:54.989610 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:54.989620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.990835 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.990839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:54.990950 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:54.990972 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:53:54.991036 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:53:54.991110 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":73,"thread_start_us":66,"threads_started":1}
I20260926 14:53:54.991581 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:54.984747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48736 (local address 127.12.68.193:40545)
0926 14:53:54.986481 (+ 1734us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:54.986491 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:54.986509 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:53:54.986565 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:54.986569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:54.986691 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:54.986789 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:54.987024 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.988152 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.989747 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:54.990554 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:54.990654 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:54.990773 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:54.991367 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 14:53:54.991455 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1643,"thread_start_us":95,"threads_started":1}
I20260926 14:53:54.992756 29849 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0a6f8571164a58a87bcd95efeadb6d (DEFAULT_TABLE table=test-table [id=97c22dc2d97d48f3ba9523a6347140ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:54.993809 29849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0a6f8571164a58a87bcd95efeadb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:54.993657 29729 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0a6f8571164a58a87bcd95efeadb6d (DEFAULT_TABLE table=test-table [id=97c22dc2d97d48f3ba9523a6347140ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:54.993683 29997 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0a6f8571164a58a87bcd95efeadb6d (DEFAULT_TABLE table=test-table [id=97c22dc2d97d48f3ba9523a6347140ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:54.994388 29729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0a6f8571164a58a87bcd95efeadb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:54.994573 29997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0a6f8571164a58a87bcd95efeadb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:54.999830 30080 tablet_bootstrap.cc:492] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a: Bootstrap starting.
I20260926 14:53:55.001255 30081 tablet_bootstrap.cc:492] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f: Bootstrap starting.
I20260926 14:53:55.002388 30080 tablet_bootstrap.cc:654] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:55.002848 30081 tablet_bootstrap.cc:654] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:55.003377 30080 log.cc:824] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:55.003815 30081 log.cc:824] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:55.003945 30082 tablet_bootstrap.cc:492] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72: Bootstrap starting.
I20260926 14:53:55.005178 30080 tablet_bootstrap.cc:492] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a: No bootstrap required, opened a new log
I20260926 14:53:55.005355 30080 ts_tablet_manager.cc:1397] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 14:53:55.005635 30081 tablet_bootstrap.cc:492] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f: No bootstrap required, opened a new log
I20260926 14:53:55.005792 30081 ts_tablet_manager.cc:1397] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 14:53:55.005949 30082 tablet_bootstrap.cc:654] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:55.006844 30082 log.cc:824] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:55.008318 30082 tablet_bootstrap.cc:492] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72: No bootstrap required, opened a new log
I20260926 14:53:55.008472 30082 ts_tablet_manager.cc:1397] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:53:55.009079 30080 raft_consensus.cc:348] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "496dbffab5e24c8d9856bf3d82ede24a" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 40545 } } peers { permanent_uuid: "228f789acd5f4488bf9c8c6138daac72" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44997 } } peers { permanent_uuid: "6fd3cc95fa774471a7d61eea86ed6e0f" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38065 } }
I20260926 14:53:55.009346 30080 raft_consensus.cc:374] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:55.009405 30080 raft_consensus.cc:729] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 496dbffab5e24c8d9856bf3d82ede24a, State: Initialized, Role: FOLLOWER
I20260926 14:53:55.009280 30081 raft_consensus.cc:348] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "496dbffab5e24c8d9856bf3d82ede24a" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 40545 } } peers { permanent_uuid: "228f789acd5f4488bf9c8c6138daac72" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44997 } } peers { permanent_uuid: "6fd3cc95fa774471a7d61eea86ed6e0f" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38065 } }
I20260926 14:53:55.009487 30081 raft_consensus.cc:374] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:55.009536 30081 raft_consensus.cc:729] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd3cc95fa774471a7d61eea86ed6e0f, State: Initialized, Role: FOLLOWER
I20260926 14:53:55.009831 30080 consensus_queue.cc:260] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "496dbffab5e24c8d9856bf3d82ede24a" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 40545 } } peers { permanent_uuid: "228f789acd5f4488bf9c8c6138daac72" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44997 } } peers { permanent_uuid: "6fd3cc95fa774471a7d61eea86ed6e0f" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38065 } }
I20260926 14:53:55.009863 30081 consensus_queue.cc:260] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "496dbffab5e24c8d9856bf3d82ede24a" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 40545 } } peers { permanent_uuid: "228f789acd5f4488bf9c8c6138daac72" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44997 } } peers { permanent_uuid: "6fd3cc95fa774471a7d61eea86ed6e0f" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38065 } }
I20260926 14:53:55.010758 30080 ts_tablet_manager.cc:1428] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260926 14:53:55.010758 30081 ts_tablet_manager.cc:1428] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260926 14:53:55.011128 29929 heartbeater.cc:503] Master 127.12.68.254:37355 was elected leader, sending a full tablet report...
I20260926 14:53:55.011678 30082 raft_consensus.cc:348] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "496dbffab5e24c8d9856bf3d82ede24a" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 40545 } } peers { permanent_uuid: "228f789acd5f4488bf9c8c6138daac72" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44997 } } peers { permanent_uuid: "6fd3cc95fa774471a7d61eea86ed6e0f" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38065 } }
I20260926 14:53:55.011916 30082 raft_consensus.cc:374] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:55.011972 30082 raft_consensus.cc:729] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 228f789acd5f4488bf9c8c6138daac72, State: Initialized, Role: FOLLOWER
I20260926 14:53:55.012027 29795 heartbeater.cc:503] Master 127.12.68.254:37355 was elected leader, sending a full tablet report...
I20260926 14:53:55.012367 30082 consensus_queue.cc:260] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "496dbffab5e24c8d9856bf3d82ede24a" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 40545 } } peers { permanent_uuid: "228f789acd5f4488bf9c8c6138daac72" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44997 } } peers { permanent_uuid: "6fd3cc95fa774471a7d61eea86ed6e0f" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38065 } }
I20260926 14:53:55.013175 30063 heartbeater.cc:503] Master 127.12.68.254:37355 was elected leader, sending a full tablet report...
I20260926 14:53:55.013973 30082 ts_tablet_manager.cc:1428] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:53:55.029160 30087 raft_consensus.cc:482] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:53:55.029299 30087 raft_consensus.cc:504] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "496dbffab5e24c8d9856bf3d82ede24a" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 40545 } } peers { permanent_uuid: "228f789acd5f4488bf9c8c6138daac72" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44997 } } peers { permanent_uuid: "6fd3cc95fa774471a7d61eea86ed6e0f" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38065 } }
I20260926 14:53:55.030267 30087 leader_election.cc:288] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 228f789acd5f4488bf9c8c6138daac72 (127.12.68.195:44997), 6fd3cc95fa774471a7d61eea86ed6e0f (127.12.68.194:38065)
I20260926 14:53:55.033406 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.030575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:58515 (local address 127.12.68.195:44997)
0926 14:53:55.030692 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.030696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.030706 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:53:55.030741 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.030743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.030777 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.030835 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:55.031088 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.031563 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.032555 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.032931 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.033029 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.033080 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.033236 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.033298 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:53:55.033406 29797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.030422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:44997 (local address 127.12.68.193:58515)
0926 14:53:55.030516 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.030530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.030606 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.030859 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.030862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.030892 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:55.030996 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.031001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.031667 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.031670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.032448 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.032457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.033031 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.033035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.033136 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.033153 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.033214 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.033281 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 14:53:55.033631 30089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.030463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:38065 (local address 127.12.68.193:35033)
0926 14:53:55.030758 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.030768 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.030863 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.031092 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.031095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.031111 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:55.031222 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.031227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.031871 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.031874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.032656 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.032665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.033258 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.033261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.033357 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.033377 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.033436 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.033498 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":78,"threads_started":1}
I20260926 14:53:55.033640 30078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.030526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:35033 (local address 127.12.68.194:38065)
0926 14:53:55.030644 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.030649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.030841 (+ 192us) server_negotiation.cc:407] Connection header received
0926 14:53:55.030966 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.030969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.031014 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.031068 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:55.031311 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.031776 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.032763 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.033135 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.033273 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.033319 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.033468 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.033529 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 14:53:55.033959 30017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0a6f8571164a58a87bcd95efeadb6d" candidate_uuid: "496dbffab5e24c8d9856bf3d82ede24a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "228f789acd5f4488bf9c8c6138daac72" is_pre_election: true
I20260926 14:53:55.034092 29883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0a6f8571164a58a87bcd95efeadb6d" candidate_uuid: "496dbffab5e24c8d9856bf3d82ede24a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd3cc95fa774471a7d61eea86ed6e0f" is_pre_election: true
I20260926 14:53:55.034204 30017 raft_consensus.cc:2454] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 496dbffab5e24c8d9856bf3d82ede24a in term 0.
I20260926 14:53:55.034333 29883 raft_consensus.cc:2454] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 496dbffab5e24c8d9856bf3d82ede24a in term 0.
I20260926 14:53:55.034616 29683 leader_election.cc:302] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 228f789acd5f4488bf9c8c6138daac72, 496dbffab5e24c8d9856bf3d82ede24a; no voters:
I20260926 14:53:55.034828 30087 raft_consensus.cc:2781] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:53:55.034896 30087 raft_consensus.cc:482] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:53:55.034946 30087 raft_consensus.cc:3037] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:55.036088 30087 raft_consensus.cc:504] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "496dbffab5e24c8d9856bf3d82ede24a" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 40545 } } peers { permanent_uuid: "228f789acd5f4488bf9c8c6138daac72" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44997 } } peers { permanent_uuid: "6fd3cc95fa774471a7d61eea86ed6e0f" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38065 } }
I20260926 14:53:55.036373 30087 leader_election.cc:288] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [CANDIDATE]: Term 1 election: Requested vote from peers 228f789acd5f4488bf9c8c6138daac72 (127.12.68.195:44997), 6fd3cc95fa774471a7d61eea86ed6e0f (127.12.68.194:38065)
I20260926 14:53:55.036664 29883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0a6f8571164a58a87bcd95efeadb6d" candidate_uuid: "496dbffab5e24c8d9856bf3d82ede24a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd3cc95fa774471a7d61eea86ed6e0f"
I20260926 14:53:55.036679 30017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0a6f8571164a58a87bcd95efeadb6d" candidate_uuid: "496dbffab5e24c8d9856bf3d82ede24a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "228f789acd5f4488bf9c8c6138daac72"
I20260926 14:53:55.036756 29883 raft_consensus.cc:3037] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:55.036769 30017 raft_consensus.cc:3037] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:55.037660 29883 raft_consensus.cc:2454] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 496dbffab5e24c8d9856bf3d82ede24a in term 1.
I20260926 14:53:55.037698 30017 raft_consensus.cc:2454] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 496dbffab5e24c8d9856bf3d82ede24a in term 1.
I20260926 14:53:55.037912 29685 leader_election.cc:302] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 496dbffab5e24c8d9856bf3d82ede24a, 6fd3cc95fa774471a7d61eea86ed6e0f; no voters:
I20260926 14:53:55.038048 30087 raft_consensus.cc:2781] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:53:55.038230 30087 raft_consensus.cc:686] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [term 1 LEADER]: Becoming Leader. State: Replica: 496dbffab5e24c8d9856bf3d82ede24a, State: Running, Role: LEADER
I20260926 14:53:55.038492 30087 consensus_queue.cc:237] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "496dbffab5e24c8d9856bf3d82ede24a" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 40545 } } peers { permanent_uuid: "228f789acd5f4488bf9c8c6138daac72" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44997 } } peers { permanent_uuid: "6fd3cc95fa774471a7d61eea86ed6e0f" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38065 } }
I20260926 14:53:55.041174 29604 catalog_manager.cc:5495] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a reported cstate change: term changed from 0 to 1, leader changed from <none> to 496dbffab5e24c8d9856bf3d82ede24a (127.12.68.193). New cstate: current_term: 1 leader_uuid: "496dbffab5e24c8d9856bf3d82ede24a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "496dbffab5e24c8d9856bf3d82ede24a" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "228f789acd5f4488bf9c8c6138daac72" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fd3cc95fa774471a7d61eea86ed6e0f" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38065 } health_report { overall_health: UNKNOWN } } }
W20260926 14:53:55.043681 29796 tablet.cc:2369] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:53:55.054399 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.051556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.12.68.193:40545)
0926 14:53:55.051647 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.051651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.051661 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:53:55.051775 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.051777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.051821 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.051926 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:53:55.052082 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.052572 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.053321 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.053516 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.053575 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.053779 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:53:55.054128 (+ 349us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:53:55.054207 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.054248 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.054290 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 14:53:55.059213 30017 raft_consensus.cc:1261] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72 [term 1 FOLLOWER]: Refusing update from remote peer 496dbffab5e24c8d9856bf3d82ede24a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:53:55.059213 29883 raft_consensus.cc:1261] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f [term 1 FOLLOWER]: Refusing update from remote peer 496dbffab5e24c8d9856bf3d82ede24a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:53:55.059736 30087 consensus_queue.cc:1035] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [LEADER]: Connected to new peer: Peer: permanent_uuid: "228f789acd5f4488bf9c8c6138daac72" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:53:55.059924 30090 consensus_queue.cc:1035] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fd3cc95fa774471a7d61eea86ed6e0f" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 38065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:53:55.067548 30094 mvcc.cc:205] Tried to move back new op lower bound from 7333619445997027328 to 7333619445919895552. Current Snapshot: MvccSnapshot[applied={T|T < 7333619445997027328 or (T in {7333619445997027328})}]
I20260926 14:53:55.073701 29602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:53:55.073828 29602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:53:55.076067 29602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf8e7fcb36c2401a9711b3aa7b068c5c: Sending DeleteTablet for 3 replicas of tablet 2a0a6f8571164a58a87bcd95efeadb6d
I20260926 14:53:55.076833 29729 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0a6f8571164a58a87bcd95efeadb6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:55 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:48736
I20260926 14:53:55.076920 29997 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0a6f8571164a58a87bcd95efeadb6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:55 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:48534
I20260926 14:53:55.077297 30103 tablet_replica.cc:330] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a: stopping tablet replica
I20260926 14:53:55.077302 29849 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0a6f8571164a58a87bcd95efeadb6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:55 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:48792
I20260926 14:53:55.077679 30103 raft_consensus.cc:2229] T 2a0a6f8571164a58a87bcd95efeadb6d P 496dbffab5e24c8d9856bf3d82ede24a [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:53:55.077786 30104 tablet_replica.cc:330] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72: stopping tablet replica
I20260926 14:53:55.077919 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 29666
I20260926 14:53:55.078509 30104 raft_consensus.cc:2229] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:53:55.078789 30104 raft_consensus.cc:2258] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:55.079259 30105 tablet_replica.cc:330] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f: stopping tablet replica
I20260926 14:53:55.079509 30105 raft_consensus.cc:2229] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:53:55.079802 30105 raft_consensus.cc:2258] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:55.080437 30104 ts_tablet_manager.cc:1905] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:55.081046 30105 ts_tablet_manager.cc:1905] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:55.082938 30104 ts_tablet_manager.cc:1918] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:53:55.083029 30104 log.cc:1196] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/2a0a6f8571164a58a87bcd95efeadb6d
I20260926 14:53:55.083340 30104 ts_tablet_manager.cc:1939] T 2a0a6f8571164a58a87bcd95efeadb6d P 228f789acd5f4488bf9c8c6138daac72: Deleting consensus metadata
I20260926 14:53:55.083413 30105 ts_tablet_manager.cc:1918] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:53:55.083483 30105 log.cc:1196] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/2a0a6f8571164a58a87bcd95efeadb6d
I20260926 14:53:55.083757 30105 ts_tablet_manager.cc:1939] T 2a0a6f8571164a58a87bcd95efeadb6d P 6fd3cc95fa774471a7d61eea86ed6e0f: Deleting consensus metadata
W20260926 14:53:55.083925 29589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:40545 (error 108)
I20260926 14:53:55.084718 29589 catalog_manager.cc:4841] TS 228f789acd5f4488bf9c8c6138daac72 (127.12.68.195:44997): tablet 2a0a6f8571164a58a87bcd95efeadb6d (table test-table [id=97c22dc2d97d48f3ba9523a6347140ba]) successfully deleted
I20260926 14:53:55.084744 29591 catalog_manager.cc:4841] TS 6fd3cc95fa774471a7d61eea86ed6e0f (127.12.68.194:38065): tablet 2a0a6f8571164a58a87bcd95efeadb6d (table test-table [id=97c22dc2d97d48f3ba9523a6347140ba]) successfully deleted
I20260926 14:53:55.084868 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.084593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:40545 (local address 127.0.0.1:48756)
0926 14:53:55.084723 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.084795 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:40545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 14:53:55.084990 29589 catalog_manager.cc:4584] TS 496dbffab5e24c8d9856bf3d82ede24a (127.12.68.193:40545): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a0a6f8571164a58a87bcd95efeadb6d: Network error: Client connection negotiation failed: client connection to 127.12.68.193:40545: connect: Connection refused (error 111)
I20260926 14:53:55.085197 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 29800
I20260926 14:53:55.089875 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 29934
I20260926 14:53:55.094146 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 29572
2026-09-26T14:53:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2033 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T14:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:53:55Z Disabled control of system clock
I20260926 14:53:55.120740 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:46623
--webserver_interface=127.12.68.254
--webserver_port=0
--builtin_ntp_servers=127.12.68.212:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:46623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:53:55.223874 30110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:55.224121 30110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:55.224157 30110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:55.228083 30110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:53:55.228132 30110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:55.228144 30110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:55.228161 30110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:53:55.228190 30110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:53:55.232738 30110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:46623
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:46623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:55.233660 30110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:55.234995 30110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:55.242565 30115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:55.242586 30116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:55.242704 30110 server_base.cc:1034] running on GCE node
W20260926 14:53:55.242583 30118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:55.243347 30110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:55.244388 30110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:55.245548 30110 hybrid_clock.cc:648] HybridClock initialized: now 1790434435245524 us; error 34 us; skew 500 ppm
I20260926 14:53:55.247195 30110 webserver.cc:460] Webserver started at http://127.12.68.254:33991/ using document root <none> and password file <none>
I20260926 14:53:55.247715 30110 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:55.247777 30110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:55.247992 30110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:55.249892 30110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "c43ec5c670ef4ce69147d0cb5e1c5528"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-24mn"
server_key: "ea28b78ec16bdc4517c5d03c5892591c"
server_key_iv: "a9f7a82234c00cf038471ecc13723a73"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.250463 30110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "c43ec5c670ef4ce69147d0cb5e1c5528"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-24mn"
server_key: "ea28b78ec16bdc4517c5d03c5892591c"
server_key_iv: "a9f7a82234c00cf038471ecc13723a73"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.253878 30110 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:53:55.256063 30124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:55.257192 30110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:53:55.257304 30110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "c43ec5c670ef4ce69147d0cb5e1c5528"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-24mn"
server_key: "ea28b78ec16bdc4517c5d03c5892591c"
server_key_iv: "a9f7a82234c00cf038471ecc13723a73"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.257396 30110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:55.299969 30110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:55.300602 30110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:55.300765 30110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:55.307200 30110 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:46623
I20260926 14:53:55.307211 30176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:46623 every 8 connection(s)
I20260926 14:53:55.308097 30110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:53:55.310910 30177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:55.315845 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 30110
I20260926 14:53:55.315982 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:53:55.316191 30177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528: Bootstrap starting.
I20260926 14:53:55.316270 12563 external_mini_cluster.cc:1428] Setting key c0029da4eb41f66f3deffa1672b87336
I20260926 14:53:55.318519 30177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:55.319250 30177 log.cc:824] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:55.321019 30177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528: No bootstrap required, opened a new log
I20260926 14:53:55.324245 30177 raft_consensus.cc:348] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ec5c670ef4ce69147d0cb5e1c5528" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 46623 } }
I20260926 14:53:55.324460 30177 raft_consensus.cc:374] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:55.324512 30177 raft_consensus.cc:729] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43ec5c670ef4ce69147d0cb5e1c5528, State: Initialized, Role: FOLLOWER
I20260926 14:53:55.324755 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.317516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.12.68.254:46623)
0926 14:53:55.318078 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.318092 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.318126 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:53:55.318768 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.318791 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.319178 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.319473 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.319985 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.320917 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.321786 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.322231 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.322367 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.323096 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.323101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.323112 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:53:55.323452 (+ 340us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.323502 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:53:55.323530 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.323609 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.323738 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.324008 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":94,"threads_started":1}
I20260926 14:53:55.324954 30177 consensus_queue.cc:260] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ec5c670ef4ce69147d0cb5e1c5528" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 46623 } }
I20260926 14:53:55.325104 30177 raft_consensus.cc:388] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:53:55.325168 30177 raft_consensus.cc:482] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:53:55.325258 30177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:55.326097 30177 raft_consensus.cc:504] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ec5c670ef4ce69147d0cb5e1c5528" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 46623 } }
I20260926 14:53:55.326395 30177 leader_election.cc:302] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c43ec5c670ef4ce69147d0cb5e1c5528; no voters:
I20260926 14:53:55.326669 30177 leader_election.cc:288] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:53:55.326742 30182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:53:55.326922 30182 raft_consensus.cc:686] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [term 1 LEADER]: Becoming Leader. State: Replica: c43ec5c670ef4ce69147d0cb5e1c5528, State: Running, Role: LEADER
I20260926 14:53:55.327215 30182 consensus_queue.cc:237] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ec5c670ef4ce69147d0cb5e1c5528" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 46623 } }
I20260926 14:53:55.327699 30177 sys_catalog.cc:564] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:53:55.328655 30184 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c43ec5c670ef4ce69147d0cb5e1c5528. Latest consensus state: current_term: 1 leader_uuid: "c43ec5c670ef4ce69147d0cb5e1c5528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ec5c670ef4ce69147d0cb5e1c5528" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 46623 } } }
I20260926 14:53:55.328769 30184 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:55.329000 30183 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c43ec5c670ef4ce69147d0cb5e1c5528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ec5c670ef4ce69147d0cb5e1c5528" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 46623 } } }
I20260926 14:53:55.329095 30183 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:55.329201 30191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:53:55.332074 30191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:53:55.337337 30191 catalog_manager.cc:1347] Generated new cluster ID: 576f729728a54fae8db06266242c0a43
I20260926 14:53:55.337384 30191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:53:55.361092 30191 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:53:55.361917 30191 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:53:55.376773 30191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528: Generated new TSK 0
I20260926 14:53:55.377380 30191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:53:55.386210 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:46623
--builtin_ntp_servers=127.12.68.212:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:53:55.489969 30201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:55.490188 30201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:55.490222 30201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:55.494143 30201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:55.494187 30201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:55.494237 30201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:53:55.498687 30201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:46623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:55.499593 30201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:55.501035 30201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:55.509663 30206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:55.509666 30209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:55.509722 30201 server_base.cc:1034] running on GCE node
W20260926 14:53:55.509660 30207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:55.510179 30201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:55.510707 30201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:55.511857 30201 hybrid_clock.cc:648] HybridClock initialized: now 1790434435511831 us; error 40 us; skew 500 ppm
I20260926 14:53:55.513540 30201 webserver.cc:460] Webserver started at http://127.12.68.193:36005/ using document root <none> and password file <none>
I20260926 14:53:55.514031 30201 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:55.514086 30201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:55.514263 30201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:55.516163 30201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "d5314ac815b840d2a536ca302f431c45"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-24mn"
server_key: "a1ef5d7f93033f4f6edd1f4a182ecf73"
server_key_iv: "03aa78156aa170eb8f87f38a9e5af7ba"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.516652 30201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "d5314ac815b840d2a536ca302f431c45"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-24mn"
server_key: "a1ef5d7f93033f4f6edd1f4a182ecf73"
server_key_iv: "03aa78156aa170eb8f87f38a9e5af7ba"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.519939 30201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:53:55.522181 30215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:55.523250 30201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 14:53:55.523336 30201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "d5314ac815b840d2a536ca302f431c45"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-24mn"
server_key: "a1ef5d7f93033f4f6edd1f4a182ecf73"
server_key_iv: "03aa78156aa170eb8f87f38a9e5af7ba"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.523427 30201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:55.539996 30201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:55.540529 30201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:55.540670 30201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:55.541164 30201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:55.541975 30201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:55.542016 30201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:55.542054 30201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:55.542083 30201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:55.549567 30201 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:37423
I20260926 14:53:55.549606 30328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:37423 every 8 connection(s)
I20260926 14:53:55.550331 30201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:53:55.551412 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 30201
I20260926 14:53:55.551523 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:53:55.551817 12563 external_mini_cluster.cc:1428] Setting key 8bc57755b929156544f735603204e559
I20260926 14:53:55.553647 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:46623
--builtin_ntp_servers=127.12.68.212:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:53:55.559020 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.552002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:60591 (local address 127.12.68.254:46623)
0926 14:53:55.552141 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.552145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.553358 (+ 1213us) server_negotiation.cc:407] Connection header received
0926 14:53:55.554720 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.554724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.554775 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.554844 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.555661 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.556197 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.557261 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.557489 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.557568 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.558595 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.558599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.558602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:53:55.558615 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.558633 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:53:55.558640 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.558695 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.558828 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.558883 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:53:55.559806 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.552504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:46623 (local address 127.12.68.193:60591)
0926 14:53:55.553249 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.553287 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.554525 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.555003 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.555009 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.555096 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:55.555530 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.555541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.556305 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.556309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.557151 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.557160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.557614 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.557619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.557771 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.558356 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:53:55.558357 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:55.558394 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:53:55.558450 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:55.558706 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:55.558715 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:55.558720 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.558819 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.559074 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":136,"threads_started":1}
I20260926 14:53:55.560890 30329 heartbeater.cc:347] Connected to a master server at 127.12.68.254:46623
I20260926 14:53:55.561095 30329 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:55.561548 30329 heartbeater.cc:511] Master 127.12.68.254:46623 requested a full tablet report, sending...
I20260926 14:53:55.562865 30141 ts_manager.cc:194] Registered new tserver with Master: d5314ac815b840d2a536ca302f431c45 (127.12.68.193:37423)
I20260926 14:53:55.563958 30141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.68.193:60591
W20260926 14:53:55.657222 30332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:55.657444 30332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:55.657492 30332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:55.661391 30332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:55.661443 30332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:55.661520 30332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:53:55.666016 30332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:46623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:55.666895 30332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:55.668180 30332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:55.676939 30338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:55.677021 30332 server_base.cc:1034] running on GCE node
W20260926 14:53:55.676945 30340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:55.676966 30337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:55.677495 30332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:55.678042 30332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:55.679211 30332 hybrid_clock.cc:648] HybridClock initialized: now 1790434435679193 us; error 28 us; skew 500 ppm
I20260926 14:53:55.680871 30332 webserver.cc:460] Webserver started at http://127.12.68.194:41287/ using document root <none> and password file <none>
I20260926 14:53:55.681370 30332 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:55.681429 30332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:55.681608 30332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:55.683470 30332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "d168e561a2534a3f97717007498d90d7"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-24mn"
server_key: "4f9e6fa604fb8d78e10aecff80501d9a"
server_key_iv: "d1153918e3768c81b3926e6d0018feae"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.684003 30332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "d168e561a2534a3f97717007498d90d7"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-24mn"
server_key: "4f9e6fa604fb8d78e10aecff80501d9a"
server_key_iv: "d1153918e3768c81b3926e6d0018feae"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.687289 30332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:53:55.689513 30346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:55.690624 30332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 14:53:55.690721 30332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "d168e561a2534a3f97717007498d90d7"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-24mn"
server_key: "4f9e6fa604fb8d78e10aecff80501d9a"
server_key_iv: "d1153918e3768c81b3926e6d0018feae"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.690804 30332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:55.710362 30332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:55.710961 30332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:55.711114 30332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:55.711606 30332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:55.712476 30332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:55.712520 30332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:55.712569 30332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:55.712597 30332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:55.720402 30332 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:45251
I20260926 14:53:55.720430 30459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:45251 every 8 connection(s)
I20260926 14:53:55.721194 30332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:53:55.728960 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 30332
I20260926 14:53:55.729076 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:53:55.729408 12563 external_mini_cluster.cc:1428] Setting key 65b4458c2ed1a752cb20c6d5aa7a37b0
I20260926 14:53:55.729835 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.722810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:33931 (local address 127.12.68.254:46623)
0926 14:53:55.722921 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.722925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.723859 (+ 934us) server_negotiation.cc:407] Connection header received
0926 14:53:55.725207 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.725211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.725268 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.725335 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.726222 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.726799 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.727987 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.728243 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.728335 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.729370 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.729373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.729376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:53:55.729387 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.729407 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:53:55.729415 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.729506 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.729638 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.729695 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:53:55.730624 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.723138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:46623 (local address 127.12.68.194:33931)
0926 14:53:55.723724 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.723760 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.725000 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.725504 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.725511 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.725590 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:55.726090 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.726102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.726916 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.726919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.727834 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.727844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.728378 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.728382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.728536 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.729045 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:53:55.729047 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:55.729106 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:53:55.729227 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:55.729508 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:55.729518 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:55.729524 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.729630 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.729875 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":100,"threads_started":1}
I20260926 14:53:55.731171 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:46623
--builtin_ntp_servers=127.12.68.212:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:53:55.731653 30460 heartbeater.cc:347] Connected to a master server at 127.12.68.254:46623
I20260926 14:53:55.731868 30460 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:55.732410 30460 heartbeater.cc:511] Master 127.12.68.254:46623 requested a full tablet report, sending...
I20260926 14:53:55.733359 30141 ts_manager.cc:194] Registered new tserver with Master: d168e561a2534a3f97717007498d90d7 (127.12.68.194:45251)
I20260926 14:53:55.733863 30141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.68.194:33931
W20260926 14:53:55.834929 30463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:55.835158 30463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:55.835208 30463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:55.839110 30463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:55.839157 30463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:55.839227 30463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:53:55.843814 30463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:46623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:55.844751 30463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:55.845976 30463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:53:55.854923 30463 server_base.cc:1034] running on GCE node
W20260926 14:53:55.854872 30468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:55.854888 30471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:55.854854 30469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:55.855439 30463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:55.856060 30463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:55.857210 30463 hybrid_clock.cc:648] HybridClock initialized: now 1790434435857192 us; error 32 us; skew 500 ppm
I20260926 14:53:55.858902 30463 webserver.cc:460] Webserver started at http://127.12.68.195:35821/ using document root <none> and password file <none>
I20260926 14:53:55.859447 30463 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:55.859510 30463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:55.859723 30463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:55.861899 30463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "f420e8e8eb344821bca2292d956f824d"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-24mn"
server_key: "1fe3f52f86501830c99839ec5dc398c3"
server_key_iv: "1a5af1226fb9133a059e59451876759f"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.862407 30463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "f420e8e8eb344821bca2292d956f824d"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-24mn"
server_key: "1fe3f52f86501830c99839ec5dc398c3"
server_key_iv: "1a5af1226fb9133a059e59451876759f"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.865782 30463 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:53:55.867945 30477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:55.869015 30463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:53:55.869127 30463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "f420e8e8eb344821bca2292d956f824d"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-24mn"
server_key: "1fe3f52f86501830c99839ec5dc398c3"
server_key_iv: "1a5af1226fb9133a059e59451876759f"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.869208 30463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:55.892057 30463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:55.892648 30463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:55.892798 30463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:55.893306 30463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:55.894156 30463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:55.894200 30463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:55.894249 30463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:55.894279 30463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:55.902287 30463 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:44373
I20260926 14:53:55.902314 30590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:44373 every 8 connection(s)
I20260926 14:53:55.903079 30463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:53:55.906728 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 30463
I20260926 14:53:55.906855 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:53:55.907150 12563 external_mini_cluster.cc:1428] Setting key 35c9df05ac7a321ae3b213c677e9b2e9
I20260926 14:53:55.911764 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.904686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:37231 (local address 127.12.68.254:46623)
0926 14:53:55.904837 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.904842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.905800 (+ 958us) server_negotiation.cc:407] Connection header received
0926 14:53:55.907321 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.907323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.907373 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.907447 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.908433 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.908953 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.910090 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.910306 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.910391 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.911337 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.911341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.911344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:53:55.911355 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.911373 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:53:55.911380 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.911437 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.911568 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.911626 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:53:55.912551 30593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.905075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:46623 (local address 127.12.68.195:37231)
0926 14:53:55.905669 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.905708 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.907065 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.907605 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.907612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.907693 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:55.908300 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.908316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.909071 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.909074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.909971 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.909981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.910433 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.910437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.910604 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.911108 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:53:55.911109 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:55.911146 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:53:55.911204 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:55.911446 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:55.911455 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:55.911460 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.911562 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.911835 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":98,"threads_started":1}
I20260926 14:53:55.913481 30591 heartbeater.cc:347] Connected to a master server at 127.12.68.254:46623
I20260926 14:53:55.913681 30591 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:55.914155 30591 heartbeater.cc:511] Master 127.12.68.254:46623 requested a full tablet report, sending...
I20260926 14:53:55.915057 30141 ts_manager.cc:194] Registered new tserver with Master: f420e8e8eb344821bca2292d956f824d (127.12.68.195:44373)
I20260926 14:53:55.915565 30141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.68.195:37231
I20260926 14:53:55.920570 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:53:55.925420 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.922316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39324 (local address 127.12.68.254:46623)
0926 14:53:55.922427 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.922431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.922575 (+ 144us) server_negotiation.cc:407] Connection header received
0926 14:53:55.922658 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.922660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.922696 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.922755 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.922957 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.923441 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.924222 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.924420 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.924491 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.925117 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.925120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.925121 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:53:55.925132 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.925148 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:53:55.925155 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.925205 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.925264 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.925305 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:53:55.928025 30141 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:53:55.930956 30141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:53:55.944595 30602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.938635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:45251 (local address 127.0.0.1:45056)
0926 14:53:55.939172 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.939185 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.939270 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.940238 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.940244 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.940265 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:55.940390 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.940397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.941856 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.941859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.943262 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.943268 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.944051 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.944054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.944195 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.944235 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.944328 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.944440 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":102,"thread_start_us":103,"threads_started":1}
I20260926 14:53:55.944643 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.938819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.12.68.194:45251)
0926 14:53:55.939771 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.939780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.939800 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:53:55.939871 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.939876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.940095 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.940244 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:55.940514 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.941742 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.943391 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.943928 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.944194 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.944372 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.944420 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.944508 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":159,"threads_started":1}
I20260926 14:53:55.944964 30606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.939364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.12.68.195:44373)
0926 14:53:55.940519 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.940530 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.940550 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:53:55.940617 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.940621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.940753 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.940866 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:55.941067 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.941787 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.943375 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.944124 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.944214 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.944348 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.944599 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.944820 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1053,"thread_start_us":124,"threads_started":1}
I20260926 14:53:55.945011 30605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.939209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:44373 (local address 127.0.0.1:41000)
0926 14:53:55.939419 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.939430 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.939503 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.940864 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.940867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.940882 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:55.940964 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.940979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.941899 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.941902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.943256 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.943266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.944399 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.944401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.944466 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.944482 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.944545 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.944613 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":119,"thread_start_us":86,"threads_started":1}
I20260926 14:53:55.945209 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.938526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:37423 (local address 127.0.0.1:50000)
0926 14:53:55.938979 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.939024 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.939138 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.940386 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.940389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.940406 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:55.940499 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.940506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.941875 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.941878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.943058 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.943067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.944041 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.944045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.944947 (+ 902us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.944968 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.945034 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.945096 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":71,"thread_start_us":81,"threads_started":1}
I20260926 14:53:55.945631 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.939568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.12.68.193:37423)
0926 14:53:55.939972 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.939981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.940000 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:53:55.940072 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.940076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.940256 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.940395 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:55.940613 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.941765 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.943166 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.943916 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.944017 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.944206 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.945413 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.945514 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":113,"threads_started":1}
I20260926 14:53:55.947444 30525 tablet_service.cc:1467] Processing CreateTablet for tablet 79317e1732e14851ab339651e9f57cb2 (DEFAULT_TABLE table=test-table [id=ebe378ffe6664ab890c30b186122eb9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:55.947489 30394 tablet_service.cc:1467] Processing CreateTablet for tablet 79317e1732e14851ab339651e9f57cb2 (DEFAULT_TABLE table=test-table [id=ebe378ffe6664ab890c30b186122eb9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:55.948015 30263 tablet_service.cc:1467] Processing CreateTablet for tablet 79317e1732e14851ab339651e9f57cb2 (DEFAULT_TABLE table=test-table [id=ebe378ffe6664ab890c30b186122eb9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:55.948441 30525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79317e1732e14851ab339651e9f57cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:55.948634 30394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79317e1732e14851ab339651e9f57cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:55.948877 30263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79317e1732e14851ab339651e9f57cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:55.954070 30608 tablet_bootstrap.cc:492] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d: Bootstrap starting.
I20260926 14:53:55.955667 30610 tablet_bootstrap.cc:492] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45: Bootstrap starting.
I20260926 14:53:55.956195 30608 tablet_bootstrap.cc:654] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:55.956187 30609 tablet_bootstrap.cc:492] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7: Bootstrap starting.
I20260926 14:53:55.957114 30608 log.cc:824] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:55.957666 30610 tablet_bootstrap.cc:654] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:55.958032 30609 tablet_bootstrap.cc:654] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:55.958438 30610 log.cc:824] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:55.958700 30608 tablet_bootstrap.cc:492] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d: No bootstrap required, opened a new log
I20260926 14:53:55.958779 30609 log.cc:824] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:55.958866 30608 ts_tablet_manager.cc:1397] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:53:55.960244 30609 tablet_bootstrap.cc:492] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7: No bootstrap required, opened a new log
I20260926 14:53:55.960391 30609 ts_tablet_manager.cc:1397] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260926 14:53:55.961946 30608 raft_consensus.cc:348] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5314ac815b840d2a536ca302f431c45" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37423 } } peers { permanent_uuid: "d168e561a2534a3f97717007498d90d7" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45251 } } peers { permanent_uuid: "f420e8e8eb344821bca2292d956f824d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44373 } }
I20260926 14:53:55.962173 30608 raft_consensus.cc:374] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:55.962237 30608 raft_consensus.cc:729] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f420e8e8eb344821bca2292d956f824d, State: Initialized, Role: FOLLOWER
I20260926 14:53:55.962697 30608 consensus_queue.cc:260] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5314ac815b840d2a536ca302f431c45" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37423 } } peers { permanent_uuid: "d168e561a2534a3f97717007498d90d7" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45251 } } peers { permanent_uuid: "f420e8e8eb344821bca2292d956f824d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44373 } }
I20260926 14:53:55.963275 30591 heartbeater.cc:503] Master 127.12.68.254:46623 was elected leader, sending a full tablet report...
I20260926 14:53:55.963523 30608 ts_tablet_manager.cc:1428] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:53:55.963762 30609 raft_consensus.cc:348] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5314ac815b840d2a536ca302f431c45" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37423 } } peers { permanent_uuid: "d168e561a2534a3f97717007498d90d7" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45251 } } peers { permanent_uuid: "f420e8e8eb344821bca2292d956f824d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44373 } }
I20260926 14:53:55.964023 30609 raft_consensus.cc:374] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:55.964083 30609 raft_consensus.cc:729] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d168e561a2534a3f97717007498d90d7, State: Initialized, Role: FOLLOWER
I20260926 14:53:55.964453 30609 consensus_queue.cc:260] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5314ac815b840d2a536ca302f431c45" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37423 } } peers { permanent_uuid: "d168e561a2534a3f97717007498d90d7" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45251 } } peers { permanent_uuid: "f420e8e8eb344821bca2292d956f824d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44373 } }
I20260926 14:53:55.964974 30460 heartbeater.cc:503] Master 127.12.68.254:46623 was elected leader, sending a full tablet report...
I20260926 14:53:55.965260 30609 ts_tablet_manager.cc:1428] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:53:55.965636 30610 tablet_bootstrap.cc:492] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45: No bootstrap required, opened a new log
I20260926 14:53:55.965808 30610 ts_tablet_manager.cc:1397] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260926 14:53:55.969003 30610 raft_consensus.cc:348] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5314ac815b840d2a536ca302f431c45" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37423 } } peers { permanent_uuid: "d168e561a2534a3f97717007498d90d7" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45251 } } peers { permanent_uuid: "f420e8e8eb344821bca2292d956f824d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44373 } }
I20260926 14:53:55.969240 30610 raft_consensus.cc:374] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:55.969296 30610 raft_consensus.cc:729] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5314ac815b840d2a536ca302f431c45, State: Initialized, Role: FOLLOWER
I20260926 14:53:55.969612 30610 consensus_queue.cc:260] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5314ac815b840d2a536ca302f431c45" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37423 } } peers { permanent_uuid: "d168e561a2534a3f97717007498d90d7" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45251 } } peers { permanent_uuid: "f420e8e8eb344821bca2292d956f824d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44373 } }
I20260926 14:53:55.970112 30329 heartbeater.cc:503] Master 127.12.68.254:46623 was elected leader, sending a full tablet report...
I20260926 14:53:55.970352 30610 ts_tablet_manager.cc:1428] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260926 14:53:55.971982 30461 tablet.cc:2369] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:53:56.051391 30330 tablet.cc:2369] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:53:56.145184 30615 raft_consensus.cc:482] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:53:56.145388 30615 raft_consensus.cc:504] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5314ac815b840d2a536ca302f431c45" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37423 } } peers { permanent_uuid: "d168e561a2534a3f97717007498d90d7" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45251 } } peers { permanent_uuid: "f420e8e8eb344821bca2292d956f824d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44373 } }
I20260926 14:53:56.146497 30615 leader_election.cc:288] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5314ac815b840d2a536ca302f431c45 (127.12.68.193:37423), f420e8e8eb344821bca2292d956f824d (127.12.68.195:44373)
I20260926 14:53:56.151134 30606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:56.146879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:49091 (local address 127.12.68.195:44373)
0926 14:53:56.146997 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:56.147001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:56.147015 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:53:56.147137 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:56.147140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:56.147184 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:56.147247 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:56.147524 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:56.148299 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:56.149845 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:56.150474 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:56.150593 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:56.150642 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:56.150924 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 14:53:56.151006 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:53:56.151134 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:56.146964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:43567 (local address 127.12.68.193:37423)
0926 14:53:56.147096 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:56.147101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:56.147282 (+ 181us) server_negotiation.cc:407] Connection header received
0926 14:53:56.147384 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:56.147386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:56.147425 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:56.147485 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:56.147723 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:56.148448 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:56.149845 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:56.150469 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:56.150594 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:56.150642 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:56.150960 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 14:53:56.151032 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:53:56.151180 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:56.146753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:44373 (local address 127.12.68.194:49091)
0926 14:53:56.146874 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:56.146889 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:56.146999 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:56.147287 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:56.147290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:56.147302 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:56.147420 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:56.147426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:56.148422 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:56.148425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:56.149700 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:56.149712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:56.150591 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:56.150594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:56.150771 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:56.150816 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 14:53:56.150884 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:53:56.150989 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":69}
I20260926 14:53:56.151347 30617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:56.146755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:37423 (local address 127.12.68.194:43567)
0926 14:53:56.147195 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:56.147206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:56.147295 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:56.147518 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:56.147521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:56.147537 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:56.147626 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:56.147630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:56.148551 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:56.148554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:56.149710 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:56.149716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:56.150600 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:56.150603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:56.150775 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:56.150811 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 14:53:56.150884 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:53:56.150989 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":137,"thread_start_us":113,"threads_started":1}
I20260926 14:53:56.151929 30545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79317e1732e14851ab339651e9f57cb2" candidate_uuid: "d168e561a2534a3f97717007498d90d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f420e8e8eb344821bca2292d956f824d" is_pre_election: true
I20260926 14:53:56.152078 30283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79317e1732e14851ab339651e9f57cb2" candidate_uuid: "d168e561a2534a3f97717007498d90d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5314ac815b840d2a536ca302f431c45" is_pre_election: true
I20260926 14:53:56.152275 30545 raft_consensus.cc:2454] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d168e561a2534a3f97717007498d90d7 in term 0.
I20260926 14:53:56.152303 30283 raft_consensus.cc:2454] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d168e561a2534a3f97717007498d90d7 in term 0.
I20260926 14:53:56.152733 30349 leader_election.cc:302] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d168e561a2534a3f97717007498d90d7, f420e8e8eb344821bca2292d956f824d; no voters:
I20260926 14:53:56.152972 30615 raft_consensus.cc:2781] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:53:56.153043 30615 raft_consensus.cc:482] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:53:56.153071 30615 raft_consensus.cc:3037] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [term 0 FOLLOWER]: Advancing to term 1
W20260926 14:53:56.153909 30592 tablet.cc:2369] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:53:56.154119 30615 raft_consensus.cc:504] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5314ac815b840d2a536ca302f431c45" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37423 } } peers { permanent_uuid: "d168e561a2534a3f97717007498d90d7" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45251 } } peers { permanent_uuid: "f420e8e8eb344821bca2292d956f824d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44373 } }
I20260926 14:53:56.154469 30615 leader_election.cc:288] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [CANDIDATE]: Term 1 election: Requested vote from peers d5314ac815b840d2a536ca302f431c45 (127.12.68.193:37423), f420e8e8eb344821bca2292d956f824d (127.12.68.195:44373)
I20260926 14:53:56.154747 30545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79317e1732e14851ab339651e9f57cb2" candidate_uuid: "d168e561a2534a3f97717007498d90d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f420e8e8eb344821bca2292d956f824d"
I20260926 14:53:56.154747 30283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79317e1732e14851ab339651e9f57cb2" candidate_uuid: "d168e561a2534a3f97717007498d90d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5314ac815b840d2a536ca302f431c45"
I20260926 14:53:56.154860 30545 raft_consensus.cc:3037] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:56.154861 30283 raft_consensus.cc:3037] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:56.156041 30283 raft_consensus.cc:2454] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d168e561a2534a3f97717007498d90d7 in term 1.
I20260926 14:53:56.156041 30545 raft_consensus.cc:2454] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d168e561a2534a3f97717007498d90d7 in term 1.
I20260926 14:53:56.156304 30349 leader_election.cc:302] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d168e561a2534a3f97717007498d90d7, f420e8e8eb344821bca2292d956f824d; no voters:
I20260926 14:53:56.156448 30615 raft_consensus.cc:2781] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:53:56.156649 30615 raft_consensus.cc:686] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [term 1 LEADER]: Becoming Leader. State: Replica: d168e561a2534a3f97717007498d90d7, State: Running, Role: LEADER
I20260926 14:53:56.156968 30615 consensus_queue.cc:237] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5314ac815b840d2a536ca302f431c45" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37423 } } peers { permanent_uuid: "d168e561a2534a3f97717007498d90d7" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45251 } } peers { permanent_uuid: "f420e8e8eb344821bca2292d956f824d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44373 } }
I20260926 14:53:56.160030 30141 catalog_manager.cc:5495] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d168e561a2534a3f97717007498d90d7 (127.12.68.194). New cstate: current_term: 1 leader_uuid: "d168e561a2534a3f97717007498d90d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5314ac815b840d2a536ca302f431c45" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d168e561a2534a3f97717007498d90d7" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f420e8e8eb344821bca2292d956f824d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44373 } health_report { overall_health: UNKNOWN } } }
I20260926 14:53:56.177698 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:56.174357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.12.68.194:45251)
0926 14:53:56.174468 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:56.174472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:56.174485 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:53:56.174619 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:56.174622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:56.174677 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:56.174757 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:53:56.174982 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:56.175474 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:56.176344 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:56.176575 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:56.176658 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:56.176934 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:53:56.177319 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:53:56.177425 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:56.177492 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 14:53:56.177542 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:53:56.183039 30545 raft_consensus.cc:1261] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d [term 1 FOLLOWER]: Refusing update from remote peer d168e561a2534a3f97717007498d90d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:53:56.183038 30283 raft_consensus.cc:1261] T 79317e1732e14851ab339651e9f57cb2 P d5314ac815b840d2a536ca302f431c45 [term 1 FOLLOWER]: Refusing update from remote peer d168e561a2534a3f97717007498d90d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:53:56.183648 30615 consensus_queue.cc:1035] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f420e8e8eb344821bca2292d956f824d" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 44373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:53:56.183830 30618 consensus_queue.cc:1035] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5314ac815b840d2a536ca302f431c45" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 37423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:53:56.189272 30621 mvcc.cc:205] Tried to move back new op lower bound from 7333619450599034880 to 7333619450501378048. Current Snapshot: MvccSnapshot[applied={T|T < 7333619450599034880}]
I20260926 14:53:56.198137 30140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:53:56.198298 30140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:53:56.200457 30140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c43ec5c670ef4ce69147d0cb5e1c5528: Sending DeleteTablet for 3 replicas of tablet 79317e1732e14851ab339651e9f57cb2
I20260926 14:53:56.201097 30263 tablet_service.cc:1514] Processing DeleteTablet for tablet 79317e1732e14851ab339651e9f57cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:56 UTC) from {username='slave'} at 127.0.0.1:50000
I20260926 14:53:56.201301 30394 tablet_service.cc:1514] Processing DeleteTablet for tablet 79317e1732e14851ab339651e9f57cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:56 UTC) from {username='slave'} at 127.0.0.1:45056
I20260926 14:53:56.201539 30525 tablet_service.cc:1514] Processing DeleteTablet for tablet 79317e1732e14851ab339651e9f57cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:56 UTC) from {username='slave'} at 127.0.0.1:41000
I20260926 14:53:56.201691 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 30201
I20260926 14:53:56.201922 30633 tablet_replica.cc:330] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d: stopping tablet replica
I20260926 14:53:56.202208 30633 raft_consensus.cc:2229] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:53:56.202471 30633 raft_consensus.cc:2258] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:56.202791 30632 tablet_replica.cc:330] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7: stopping tablet replica
I20260926 14:53:56.203032 30632 raft_consensus.cc:2229] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:53:56.203311 30632 raft_consensus.cc:2258] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:56.204486 30632 ts_tablet_manager.cc:1905] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:56.204486 30633 ts_tablet_manager.cc:1905] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:56.207108 30632 ts_tablet_manager.cc:1918] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:53:56.207196 30632 log.cc:1196] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/79317e1732e14851ab339651e9f57cb2
I20260926 14:53:56.207122 30633 ts_tablet_manager.cc:1918] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:53:56.207357 30633 log.cc:1196] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/79317e1732e14851ab339651e9f57cb2
I20260926 14:53:56.207451 30632 ts_tablet_manager.cc:1939] T 79317e1732e14851ab339651e9f57cb2 P d168e561a2534a3f97717007498d90d7: Deleting consensus metadata
I20260926 14:53:56.207733 30633 ts_tablet_manager.cc:1939] T 79317e1732e14851ab339651e9f57cb2 P f420e8e8eb344821bca2292d956f824d: Deleting consensus metadata
I20260926 14:53:56.208792 30127 catalog_manager.cc:4841] TS d168e561a2534a3f97717007498d90d7 (127.12.68.194:45251): tablet 79317e1732e14851ab339651e9f57cb2 (table test-table [id=ebe378ffe6664ab890c30b186122eb9d]) successfully deleted
I20260926 14:53:56.208981 30127 catalog_manager.cc:4841] TS f420e8e8eb344821bca2292d956f824d (127.12.68.195:44373): tablet 79317e1732e14851ab339651e9f57cb2 (table test-table [id=ebe378ffe6664ab890c30b186122eb9d]) successfully deleted
W20260926 14:53:56.210109 30128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:37423 (error 108)
I20260926 14:53:56.210273 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 30332
I20260926 14:53:56.212316 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:56.212073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:37423 (local address 127.0.0.1:50002)
0926 14:53:56.212188 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:56.212246 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 14:53:56.212443 30128 catalog_manager.cc:4584] TS d5314ac815b840d2a536ca302f431c45 (127.12.68.193:37423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79317e1732e14851ab339651e9f57cb2: Network error: Client connection negotiation failed: client connection to 127.12.68.193:37423: connect: Connection refused (error 111)
I20260926 14:53:56.215075 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 30463
I20260926 14:53:56.219584 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 30110
2026-09-26T14:53:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1125 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:53:56 dist-test-slave-24mn krb5kdc[30637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:53:56 dist-test-slave-24mn krb5kdc[30637](info): set up 2 sockets
Sep 26 14:53:56 dist-test-slave-24mn krb5kdc[30637](info): commencing operation
krb5kdc: starting...
W20260926 14:53:57.171654 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:53:57 dist-test-slave-24mn krb5kdc[30637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:53:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:53:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:53:57.282027 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:37585
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:37585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:57.386816 30650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:57.387059 30650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:57.387097 30650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:57.390990 30650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:53:57.391040 30650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:57.391058 30650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:57.391075 30650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:53:57.391103 30650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:53:57.396122 30650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:37585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:37585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:57.397064 30650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:57.398002 30650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:57.405678 30655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:57.405684 30658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:57.405730 30656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:57.405926 30650 server_base.cc:1034] running on GCE node
I20260926 14:53:57.406512 30650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:57.407510 30650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:57.408654 30650 hybrid_clock.cc:648] HybridClock initialized: now 1790434437408630 us; error 37 us; skew 500 ppm
Sep 26 14:53:57 dist-test-slave-24mn krb5kdc[30637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434437, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:57.411167 30650 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:53:57.411962 30650 webserver.cc:460] Webserver started at http://127.12.68.254:36267/ using document root <none> and password file <none>
I20260926 14:53:57.412451 30650 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:57.412506 30650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:57.412699 30650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:57.414608 30650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "66d0398bd2fe4cc4bd80db3a3894e41f"
format_stamp: "Formatted at 2026-09-26 14:53:57 on dist-test-slave-24mn"
server_key: "91456917bc94981bab45acc72710b681"
server_key_iv: "3aeb955ecc967db9107e34a2cbab75d7"
server_key_version: "encryptionkey@0"
I20260926 14:53:57.415110 30650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "66d0398bd2fe4cc4bd80db3a3894e41f"
format_stamp: "Formatted at 2026-09-26 14:53:57 on dist-test-slave-24mn"
server_key: "91456917bc94981bab45acc72710b681"
server_key_iv: "3aeb955ecc967db9107e34a2cbab75d7"
server_key_version: "encryptionkey@0"
I20260926 14:53:57.418497 30650 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:53:57.420656 30665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:57.421743 30650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 14:53:57.421847 30650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "66d0398bd2fe4cc4bd80db3a3894e41f"
format_stamp: "Formatted at 2026-09-26 14:53:57 on dist-test-slave-24mn"
server_key: "91456917bc94981bab45acc72710b681"
server_key_iv: "3aeb955ecc967db9107e34a2cbab75d7"
server_key_version: "encryptionkey@0"
I20260926 14:53:57.421931 30650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:57.445770 30650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:57.448338 30650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:57.448495 30650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:57.454676 30650 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:37585
I20260926 14:53:57.454703 30717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:37585 every 8 connection(s)
I20260926 14:53:57.455503 30650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:53:57.457018 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 30650
I20260926 14:53:57.457135 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:53:57.457389 12563 external_mini_cluster.cc:1428] Setting key bb6f433d96beb231816f86ed0d3a9cab
I20260926 14:53:57.458149 30718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:57.463418 30718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f: Bootstrap starting.
Sep 26 14:53:57 dist-test-slave-24mn krb5kdc[30637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:53:57.465752 30718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:57.466542 30718 log.cc:824] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:57.468583 30718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f: No bootstrap required, opened a new log
I20260926 14:53:57.470489 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:57.458648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.12.68.254:37585)
0926 14:53:57.459042 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:57.459056 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:57.459095 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:53:57.459681 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:57.459696 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:57.460051 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:57.460354 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:57.461104 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:57.462063 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:57.462881 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:57.463334 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:57.463468 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:57.465938 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:57.465944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:57.465962 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:57.465992 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:57.467657 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:57.468097 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:57.468102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:57.468114 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:57.468245 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:57.468462 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:57.468465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:57.468467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:57.468936 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:57.469122 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:57.469326 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 14:53:57.469599 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":133,"threads_started":1}
I20260926 14:53:57.472728 30718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d0398bd2fe4cc4bd80db3a3894e41f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37585 } }
I20260926 14:53:57.472937 30718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:57.472970 30718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d0398bd2fe4cc4bd80db3a3894e41f, State: Initialized, Role: FOLLOWER
I20260926 14:53:57.473301 30718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d0398bd2fe4cc4bd80db3a3894e41f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37585 } }
I20260926 14:53:57.473404 30718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:53:57.473452 30718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:53:57.473515 30718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:57.474267 30718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d0398bd2fe4cc4bd80db3a3894e41f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37585 } }
I20260926 14:53:57.474536 30718 leader_election.cc:302] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66d0398bd2fe4cc4bd80db3a3894e41f; no voters:
I20260926 14:53:57.474769 30718 leader_election.cc:288] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:53:57.474864 30723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:53:57.475064 30723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [term 1 LEADER]: Becoming Leader. State: Replica: 66d0398bd2fe4cc4bd80db3a3894e41f, State: Running, Role: LEADER
I20260926 14:53:57.475376 30723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d0398bd2fe4cc4bd80db3a3894e41f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37585 } }
I20260926 14:53:57.475785 30718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:53:57.476876 30724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66d0398bd2fe4cc4bd80db3a3894e41f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d0398bd2fe4cc4bd80db3a3894e41f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37585 } } }
I20260926 14:53:57.476980 30724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:57.477262 30725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66d0398bd2fe4cc4bd80db3a3894e41f. Latest consensus state: current_term: 1 leader_uuid: "66d0398bd2fe4cc4bd80db3a3894e41f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d0398bd2fe4cc4bd80db3a3894e41f" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37585 } } }
I20260926 14:53:57.477402 30725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:57.477334 30732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:53:57.480854 30732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:53:57.485793 30732 catalog_manager.cc:1347] Generated new cluster ID: 85e361d57ec64f469a86b9700b3693c3
I20260926 14:53:57.485854 30732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:53:57.523190 30732 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:53:57.524493 30732 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:53:57.540205 30732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f: Generated new TSK 0
I20260926 14:53:57.540874 30732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:53:57.578655 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:37585
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:57.705799 30744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:57.706033 30744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:57.706073 30744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:57.710052 30744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:57.710105 30744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:57.710161 30744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:53:57.715262 30744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:37585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:57.716246 30744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:57.717203 30744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:57.726070 30750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:57.726101 30752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:57.726101 30749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:57.726303 30744 server_base.cc:1034] running on GCE node
I20260926 14:53:57.726754 30744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:57.727339 30744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:57.728519 30744 hybrid_clock.cc:648] HybridClock initialized: now 1790434437728492 us; error 41 us; skew 500 ppm
Sep 26 14:53:57 dist-test-slave-24mn krb5kdc[30637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434437, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:57.731168 30744 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:53:57.732033 30744 webserver.cc:460] Webserver started at http://127.12.68.193:32887/ using document root <none> and password file <none>
I20260926 14:53:57.732558 30744 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:57.732620 30744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:57.732810 30744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:57.734742 30744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "831f4b11260043758976560d7360e97c"
format_stamp: "Formatted at 2026-09-26 14:53:57 on dist-test-slave-24mn"
server_key: "dc9a28d0995e7b6a7728c5f9b71e803a"
server_key_iv: "ec2ae4652aec59b96af3202d6631ef1b"
server_key_version: "encryptionkey@0"
I20260926 14:53:57.735253 30744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "831f4b11260043758976560d7360e97c"
format_stamp: "Formatted at 2026-09-26 14:53:57 on dist-test-slave-24mn"
server_key: "dc9a28d0995e7b6a7728c5f9b71e803a"
server_key_iv: "ec2ae4652aec59b96af3202d6631ef1b"
server_key_version: "encryptionkey@0"
I20260926 14:53:57.738904 30744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:53:57.741428 30759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:57.742630 30744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:53:57.742741 30744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "831f4b11260043758976560d7360e97c"
format_stamp: "Formatted at 2026-09-26 14:53:57 on dist-test-slave-24mn"
server_key: "dc9a28d0995e7b6a7728c5f9b71e803a"
server_key_iv: "ec2ae4652aec59b96af3202d6631ef1b"
server_key_version: "encryptionkey@0"
I20260926 14:53:57.742838 30744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:57.757195 30744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:57.760099 30744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:57.760259 30744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:57.760795 30744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:57.761549 30744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:57.761595 30744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:57.761646 30744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:57.761672 30744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:57.769567 30744 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:42961
I20260926 14:53:57.769598 30872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:42961 every 8 connection(s)
I20260926 14:53:57.770381 30744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:53:57.774822 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 30744
I20260926 14:53:57.774950 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:53:57.775305 12563 external_mini_cluster.cc:1428] Setting key f6b002fab37451405d02efd39d34aa10
Sep 26 14:53:57 dist-test-slave-24mn krb5kdc[30637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434437, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
I20260926 14:53:57.784587 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:57.772030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:58175 (local address 127.12.68.254:37585)
0926 14:53:57.772198 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:57.772202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:57.773008 (+ 806us) server_negotiation.cc:407] Connection header received
0926 14:53:57.774253 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:57.774256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:57.774311 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:57.774380 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:57.775835 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:57.776548 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:57.778081 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:57.778388 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:57.778496 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:57.780897 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:57.780901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:57.780903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:57.780926 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:57.782485 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:57.782992 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:57.782995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:57.782997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:57.783090 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:57.783426 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:57.783429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:57.783439 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:57.783659 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:57.783765 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:57.784285 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 14:53:57.784408 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 14:53:57.785333 30875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:57.772410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:37585 (local address 127.12.68.193:58175)
0926 14:53:57.772847 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:57.772887 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:57.774057 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:57.774550 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:57.774558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:57.775012 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:57.775692 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:57.775712 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:57.776660 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:57.776663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:57.777971 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:57.777982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:57.778523 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:57.778526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:57.778659 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:57.779165 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:57.779181 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:57.780732 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:57.782622 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:57.782627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:57.782639 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:57.782895 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:57.783186 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:57.783189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:57.783190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:57.783329 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:57.783750 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:57.783754 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:57.784056 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 14:53:57.784211 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 14:53:57.784454 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260926 14:53:57.786602 30873 heartbeater.cc:347] Connected to a master server at 127.12.68.254:37585
I20260926 14:53:57.786842 30873 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
I20260926 14:53:57.787372 30873 heartbeater.cc:511] Master 127.12.68.254:37585 requested a full tablet report, sending...
I20260926 14:53:57.788867 30682 ts_manager.cc:194] Registered new tserver with Master: 831f4b11260043758976560d7360e97c (127.12.68.193:42961)
I20260926 14:53:57.789986 30682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:58175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:53:57.802909 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:37585
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:57.907281 30878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:57.907512 30878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:57.907557 30878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:57.911435 30878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:57.911486 30878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:57.911540 30878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:53:57.916507 30878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:37585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:57.917410 30878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:57.918332 30878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:57.927201 30886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:57.927241 30883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:57.927219 30884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:57.927408 30878 server_base.cc:1034] running on GCE node
I20260926 14:53:57.927865 30878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:57.928481 30878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:57.929625 30878 hybrid_clock.cc:648] HybridClock initialized: now 1790434437929598 us; error 41 us; skew 500 ppm
Sep 26 14:53:57 dist-test-slave-24mn krb5kdc[30637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434437, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:57.932168 30878 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:53:57.932957 30878 webserver.cc:460] Webserver started at http://127.12.68.194:45915/ using document root <none> and password file <none>
I20260926 14:53:57.933460 30878 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:57.933521 30878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:57.933696 30878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:57.935585 30878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "69dce9f1b04743e8ae9ceb0c185286b6"
format_stamp: "Formatted at 2026-09-26 14:53:57 on dist-test-slave-24mn"
server_key: "6b84e3a5bf099f400cdcaaafe48958ba"
server_key_iv: "e157428a6c2dc99bf425b4004b9efc83"
server_key_version: "encryptionkey@0"
I20260926 14:53:57.936101 30878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "69dce9f1b04743e8ae9ceb0c185286b6"
format_stamp: "Formatted at 2026-09-26 14:53:57 on dist-test-slave-24mn"
server_key: "6b84e3a5bf099f400cdcaaafe48958ba"
server_key_iv: "e157428a6c2dc99bf425b4004b9efc83"
server_key_version: "encryptionkey@0"
I20260926 14:53:57.939405 30878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 14:53:57.941635 30893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:57.942780 30878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:53:57.942888 30878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "69dce9f1b04743e8ae9ceb0c185286b6"
format_stamp: "Formatted at 2026-09-26 14:53:57 on dist-test-slave-24mn"
server_key: "6b84e3a5bf099f400cdcaaafe48958ba"
server_key_iv: "e157428a6c2dc99bf425b4004b9efc83"
server_key_version: "encryptionkey@0"
I20260926 14:53:57.942972 30878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:57.965025 30878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:57.968001 30878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:57.968169 30878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:57.968695 30878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:57.969453 30878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:57.969498 30878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:57.969548 30878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:57.969576 30878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:57.977732 30878 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:45137
I20260926 14:53:57.977801 31006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:45137 every 8 connection(s)
I20260926 14:53:57.978562 30878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:53:57 dist-test-slave-24mn krb5kdc[30637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434437, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:53:57.988415 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 30878
I20260926 14:53:57.988538 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:53:57.988792 12563 external_mini_cluster.cc:1428] Setting key 41aec98f9523b56a26f68085cea37290
I20260926 14:53:57.990520 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:57.980136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:55943 (local address 127.12.68.254:37585)
0926 14:53:57.980283 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:57.980288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:57.981046 (+ 758us) server_negotiation.cc:407] Connection header received
0926 14:53:57.982246 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:57.982248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:57.982293 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:57.982361 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:57.983393 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:57.983946 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:57.985003 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:57.985213 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:57.985277 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:57.987325 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:57.987328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:57.987330 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:57.987352 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:57.988476 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:57.988942 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:57.988946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:57.988948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:57.989071 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:57.989409 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:57.989412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:57.989414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:57.989605 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:57.989680 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:57.990244 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 14:53:57.990362 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:53:57.991214 31009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:57.980511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:37585 (local address 127.12.68.194:55943)
0926 14:53:57.980913 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:57.980952 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:57.982071 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:57.982500 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:57.982507 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:57.982831 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:57.983272 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:57.983284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:57.984049 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:57.984052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:57.984912 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:57.984920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:57.985311 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:57.985314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:57.985442 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:57.985891 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:57.985905 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:57.987184 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:57.988594 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:57.988600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:57.988614 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:57.988847 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:57.989162 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:57.989165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:57.989166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:57.989329 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:57.989703 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:57.989709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:57.989984 (+ 275us) client_negotiation.cc:769] Sending connection context
0926 14:53:57.990174 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 14:53:57.990413 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":100,"threads_started":1}
I20260926 14:53:57.992247 31007 heartbeater.cc:347] Connected to a master server at 127.12.68.254:37585
I20260926 14:53:57.992491 31007 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:57.992965 31007 heartbeater.cc:511] Master 127.12.68.254:37585 requested a full tablet report, sending...
I20260926 14:53:57.993829 30682 ts_manager.cc:194] Registered new tserver with Master: 69dce9f1b04743e8ae9ceb0c185286b6 (127.12.68.194:45137)
I20260926 14:53:57.994369 30682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:55943
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:53:58.012826 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:37585
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:58.116305 31012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:58.116539 31012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:58.116577 31012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:58.120697 31012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:58.120760 31012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:58.120839 31012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:53:58.126487 31012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:37585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:58.127413 31012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:58.128378 31012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:58.137316 31020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:58.137316 31017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:58.137293 31018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:58.137501 31012 server_base.cc:1034] running on GCE node
I20260926 14:53:58.137841 31012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:58.138381 31012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:58.139535 31012 hybrid_clock.cc:648] HybridClock initialized: now 1790434438139513 us; error 34 us; skew 500 ppm
Sep 26 14:53:58 dist-test-slave-24mn krb5kdc[30637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434438, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:58.145176 31012 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:53:58.146047 31012 webserver.cc:460] Webserver started at http://127.12.68.195:35097/ using document root <none> and password file <none>
I20260926 14:53:58.146554 31012 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:58.146613 31012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:58.146824 31012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:58.148862 31012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "6e553496354446f1a839a2a86cc57c3e"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-24mn"
server_key: "195a914b2949856df1d00ebfd672bedd"
server_key_iv: "da8c1faf9e39e6f6daad4b7ef11c329c"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.149377 31012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "6e553496354446f1a839a2a86cc57c3e"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-24mn"
server_key: "195a914b2949856df1d00ebfd672bedd"
server_key_iv: "da8c1faf9e39e6f6daad4b7ef11c329c"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.152813 31012 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:53:58.155005 31027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:58.156219 31012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:53:58.156319 31012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "6e553496354446f1a839a2a86cc57c3e"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-24mn"
server_key: "195a914b2949856df1d00ebfd672bedd"
server_key_iv: "da8c1faf9e39e6f6daad4b7ef11c329c"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.156409 31012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:58.172503 31012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:58.175230 31012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:58.175392 31012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:58.175928 31012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:58.176709 31012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:58.176761 31012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:58.176812 31012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:58.176837 31012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:58.185055 31012 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:36459
I20260926 14:53:58.185074 31140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:36459 every 8 connection(s)
I20260926 14:53:58.185861 31012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:53:58.187944 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 31012
I20260926 14:53:58.188073 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:53:58.188341 12563 external_mini_cluster.cc:1428] Setting key 3370bb610363af47dbfa2495fc5894f7
Sep 26 14:53:58 dist-test-slave-24mn krb5kdc[30637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434438, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:53:58.197791 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.187499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:59191 (local address 127.12.68.254:37585)
0926 14:53:58.187617 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:58.187621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:58.188449 (+ 828us) server_negotiation.cc:407] Connection header received
0926 14:53:58.189701 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:58.189704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:58.189747 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:58.189799 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:58.190945 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.191481 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.192568 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.192783 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.192859 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.195005 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:58.195008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:58.195010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:58.195034 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:58.196223 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:58.196628 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:58.196631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:58.196632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:58.196706 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:58.196957 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:58.196959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:58.196960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:58.197146 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:58.197218 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:58.197591 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 14:53:58.197677 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:53:58.198508 31143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.187786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:37585 (local address 127.12.68.195:59191)
0926 14:53:58.188253 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:58.188304 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:58.189496 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:58.189994 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:58.190002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:58.190360 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:58.190821 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.190834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.191571 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.191574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.192458 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.192467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.192887 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.192890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.193015 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.193511 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:58.193526 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:58.194829 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:58.196309 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:58.196313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:58.196323 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:58.196545 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:58.196772 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:58.196773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:58.196774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:58.196884 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:58.197219 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:58.197223 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:58.197421 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 14:53:58.197559 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 14:53:58.197792 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260926 14:53:58.199479 31141 heartbeater.cc:347] Connected to a master server at 127.12.68.254:37585
I20260926 14:53:58.199680 31141 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:58.200119 31141 heartbeater.cc:511] Master 127.12.68.254:37585 requested a full tablet report, sending...
I20260926 14:53:58.201025 30682 ts_manager.cc:194] Registered new tserver with Master: 6e553496354446f1a839a2a86cc57c3e (127.12.68.195:36459)
I20260926 14:53:58.201689 30682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:59191
I20260926 14:53:58.201753 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:53:58.210006 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.203603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56106 (local address 127.12.68.254:37585)
0926 14:53:58.203701 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:58.203704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:58.203868 (+ 164us) server_negotiation.cc:407] Connection header received
0926 14:53:58.203998 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:58.204000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:58.204033 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:58.204090 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:58.204665 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.205129 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.206009 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.206214 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.206285 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.207523 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:58.207526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:58.207528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:58.207547 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:58.208735 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:58.209130 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:58.209132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:58.209133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:58.209194 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:58.209412 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:58.209414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:58.209415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:58.209580 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:58.209692 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:58.209841 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 14:53:58.209905 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:53:58.212415 30682 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:53:58.215447 30682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:53:58.229218 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.223016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.12.68.193:42961)
0926 14:53:58.223334 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:58.223345 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:58.224145 (+ 800us) server_negotiation.cc:407] Connection header received
0926 14:53:58.224371 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:58.224378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:58.224559 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:58.224721 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:58.224999 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.226254 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.227724 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.228505 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.228639 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.228801 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:58.228958 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 14:53:58.229058 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":130,"threads_started":1}
I20260926 14:53:58.229238 31153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.222977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:36459 (local address 127.0.0.1:43310)
0926 14:53:58.224515 (+ 1538us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:58.224527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:58.224610 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:58.225034 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:58.225037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:58.225052 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:58.225149 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.225154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.226841 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.226845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.228011 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.228019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.228849 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.228853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.228933 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.228953 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:53:58.228991 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 14:53:58.229041 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1483,"thread_start_us":69,"threads_started":1}
I20260926 14:53:58.229529 31152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.223547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:45137 (local address 127.0.0.1:33442)
0926 14:53:58.224799 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:58.224810 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:58.224882 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:58.225357 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:58.225360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:58.225377 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:58.225460 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.225466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.226534 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.226537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.227830 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.227840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.229043 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.229047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.229148 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.229170 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:53:58.229354 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 14:53:58.229430 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"thread_start_us":84,"threads_started":1}
I20260926 14:53:58.229238 31155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.222867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:42961 (local address 127.0.0.1:34424)
0926 14:53:58.223957 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:58.224005 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:58.224160 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:58.224720 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:58.224726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:58.224755 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:58.224896 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.224903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.226381 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.226385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.227610 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.227620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.228650 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.228654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.228760 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.228808 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 14:53:58.228910 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:53:58.229029 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":193,"thread_start_us":69,"threads_started":1}
I20260926 14:53:58.229977 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.223719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33442 (local address 127.12.68.194:45137)
0926 14:53:58.224432 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:58.224442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:58.224974 (+ 532us) server_negotiation.cc:407] Connection header received
0926 14:53:58.225052 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:58.225057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:58.225244 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:58.225375 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:58.225549 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.226398 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.227973 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.228762 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.228860 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.228984 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:58.229735 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 14:53:58.229846 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":99,"threads_started":1}
I20260926 14:53:58.229981 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.223929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.12.68.195:36459)
0926 14:53:58.224394 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:58.224404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:58.224622 (+ 218us) server_negotiation.cc:407] Connection header received
0926 14:53:58.224746 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:58.224751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:58.224926 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:58.225069 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:58.225234 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.226329 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.228131 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.228722 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.229569 (+ 847us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.229706 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:58.229759 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 14:53:58.229853 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":164,"threads_started":1}
I20260926 14:53:58.232400 30807 tablet_service.cc:1467] Processing CreateTablet for tablet 07b00f00e6024d488c7dcbf7546a7df5 (DEFAULT_TABLE table=test-table [id=d81f300ea7fa4ed78ced01a825405c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:58.232398 31075 tablet_service.cc:1467] Processing CreateTablet for tablet 07b00f00e6024d488c7dcbf7546a7df5 (DEFAULT_TABLE table=test-table [id=d81f300ea7fa4ed78ced01a825405c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:58.233290 31075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07b00f00e6024d488c7dcbf7546a7df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:58.233290 30807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07b00f00e6024d488c7dcbf7546a7df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:58.236372 30941 tablet_service.cc:1467] Processing CreateTablet for tablet 07b00f00e6024d488c7dcbf7546a7df5 (DEFAULT_TABLE table=test-table [id=d81f300ea7fa4ed78ced01a825405c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:58.237366 30941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07b00f00e6024d488c7dcbf7546a7df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:58.240031 31158 tablet_bootstrap.cc:492] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c: Bootstrap starting.
I20260926 14:53:58.240446 31159 tablet_bootstrap.cc:492] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e: Bootstrap starting.
I20260926 14:53:58.242421 31158 tablet_bootstrap.cc:654] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:58.242421 31159 tablet_bootstrap.cc:654] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:58.243407 31160 tablet_bootstrap.cc:492] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6: Bootstrap starting.
I20260926 14:53:58.243656 31159 log.cc:824] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:58.243656 31158 log.cc:824] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:58.245489 31160 tablet_bootstrap.cc:654] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:58.245679 31159 tablet_bootstrap.cc:492] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e: No bootstrap required, opened a new log
I20260926 14:53:58.245687 31158 tablet_bootstrap.cc:492] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c: No bootstrap required, opened a new log
I20260926 14:53:58.245869 31159 ts_tablet_manager.cc:1397] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:53:58.245883 31158 ts_tablet_manager.cc:1397] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 14:53:58.246520 31160 log.cc:824] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:58.248202 31160 tablet_bootstrap.cc:492] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6: No bootstrap required, opened a new log
I20260926 14:53:58.248366 31160 ts_tablet_manager.cc:1397] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:53:58.249648 31158 raft_consensus.cc:348] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831f4b11260043758976560d7360e97c" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 42961 } } peers { permanent_uuid: "6e553496354446f1a839a2a86cc57c3e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36459 } } peers { permanent_uuid: "69dce9f1b04743e8ae9ceb0c185286b6" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45137 } }
I20260926 14:53:58.249650 31159 raft_consensus.cc:348] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831f4b11260043758976560d7360e97c" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 42961 } } peers { permanent_uuid: "6e553496354446f1a839a2a86cc57c3e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36459 } } peers { permanent_uuid: "69dce9f1b04743e8ae9ceb0c185286b6" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45137 } }
I20260926 14:53:58.249905 31158 raft_consensus.cc:374] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:58.249905 31159 raft_consensus.cc:374] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:58.249960 31158 raft_consensus.cc:729] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 831f4b11260043758976560d7360e97c, State: Initialized, Role: FOLLOWER
I20260926 14:53:58.249960 31159 raft_consensus.cc:729] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e553496354446f1a839a2a86cc57c3e, State: Initialized, Role: FOLLOWER
I20260926 14:53:58.250401 31159 consensus_queue.cc:260] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831f4b11260043758976560d7360e97c" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 42961 } } peers { permanent_uuid: "6e553496354446f1a839a2a86cc57c3e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36459 } } peers { permanent_uuid: "69dce9f1b04743e8ae9ceb0c185286b6" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45137 } }
I20260926 14:53:58.250401 31158 consensus_queue.cc:260] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831f4b11260043758976560d7360e97c" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 42961 } } peers { permanent_uuid: "6e553496354446f1a839a2a86cc57c3e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36459 } } peers { permanent_uuid: "69dce9f1b04743e8ae9ceb0c185286b6" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45137 } }
I20260926 14:53:58.251051 30873 heartbeater.cc:503] Master 127.12.68.254:37585 was elected leader, sending a full tablet report...
I20260926 14:53:58.251327 31141 heartbeater.cc:503] Master 127.12.68.254:37585 was elected leader, sending a full tablet report...
I20260926 14:53:58.251382 31158 ts_tablet_manager.cc:1428] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:53:58.251289 31160 raft_consensus.cc:348] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831f4b11260043758976560d7360e97c" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 42961 } } peers { permanent_uuid: "6e553496354446f1a839a2a86cc57c3e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36459 } } peers { permanent_uuid: "69dce9f1b04743e8ae9ceb0c185286b6" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45137 } }
I20260926 14:53:58.251514 31160 raft_consensus.cc:374] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:58.251560 31160 raft_consensus.cc:729] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69dce9f1b04743e8ae9ceb0c185286b6, State: Initialized, Role: FOLLOWER
I20260926 14:53:58.251973 31160 consensus_queue.cc:260] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831f4b11260043758976560d7360e97c" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 42961 } } peers { permanent_uuid: "6e553496354446f1a839a2a86cc57c3e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36459 } } peers { permanent_uuid: "69dce9f1b04743e8ae9ceb0c185286b6" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45137 } }
I20260926 14:53:58.252369 31159 ts_tablet_manager.cc:1428] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 14:53:58.252954 31007 heartbeater.cc:503] Master 127.12.68.254:37585 was elected leader, sending a full tablet report...
I20260926 14:53:58.254443 31160 ts_tablet_manager.cc:1428] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:53:58.254848 31165 raft_consensus.cc:482] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:53:58.254999 31165 raft_consensus.cc:504] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831f4b11260043758976560d7360e97c" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 42961 } } peers { permanent_uuid: "6e553496354446f1a839a2a86cc57c3e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36459 } } peers { permanent_uuid: "69dce9f1b04743e8ae9ceb0c185286b6" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45137 } }
I20260926 14:53:58.256745 31165 leader_election.cc:288] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e553496354446f1a839a2a86cc57c3e (127.12.68.195:36459), 69dce9f1b04743e8ae9ceb0c185286b6 (127.12.68.194:45137)
I20260926 14:53:58.259568 30875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.256515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:36459 (local address 127.12.68.193:50119)
0926 14:53:58.256632 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:58.256647 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:58.256724 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:58.257030 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:58.257033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:58.257050 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:58.257150 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.257156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.257822 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.257825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.258576 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.258584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.259192 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.259195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.259295 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.259317 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:53:58.259380 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:53:58.259447 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 14:53:58.259568 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.256725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:50119 (local address 127.12.68.195:36459)
0926 14:53:58.256859 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:58.256863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:58.256874 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:53:58.256912 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:58.256915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:58.256955 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:58.257017 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:58.257243 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.257727 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.258696 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.259085 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.259192 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.259239 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:58.259400 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 14:53:58.259462 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 14:53:58.260095 31095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07b00f00e6024d488c7dcbf7546a7df5" candidate_uuid: "831f4b11260043758976560d7360e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e553496354446f1a839a2a86cc57c3e" is_pre_election: true
I20260926 14:53:58.260253 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.257005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:52529 (local address 127.12.68.194:45137)
0926 14:53:58.257111 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:58.257115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:58.257329 (+ 214us) server_negotiation.cc:407] Connection header received
0926 14:53:58.257428 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:58.257431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:58.257469 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:58.257523 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:58.257765 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.258234 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.259198 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.259661 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.259771 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.259812 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:58.260091 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 14:53:58.260154 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:53:58.260272 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.256925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:45137 (local address 127.12.68.193:52529)
0926 14:53:58.257249 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:58.257259 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:58.257345 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:58.257537 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:58.257539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:58.257555 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:58.257676 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.257682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.258332 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.258336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.259091 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.259099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.259865 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.259869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.259988 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.260010 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:53:58.260076 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 14:53:58.260144 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":89,"threads_started":1}
I20260926 14:53:58.260327 31095 raft_consensus.cc:2454] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831f4b11260043758976560d7360e97c in term 0.
I20260926 14:53:58.260775 30762 leader_election.cc:302] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e553496354446f1a839a2a86cc57c3e, 831f4b11260043758976560d7360e97c; no voters:
I20260926 14:53:58.260998 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07b00f00e6024d488c7dcbf7546a7df5" candidate_uuid: "831f4b11260043758976560d7360e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69dce9f1b04743e8ae9ceb0c185286b6" is_pre_election: true
I20260926 14:53:58.261037 31165 raft_consensus.cc:2781] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:53:58.261107 31165 raft_consensus.cc:482] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:53:58.261140 31165 raft_consensus.cc:3037] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:58.261238 30961 raft_consensus.cc:2454] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831f4b11260043758976560d7360e97c in term 0.
I20260926 14:53:58.262246 31165 raft_consensus.cc:504] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831f4b11260043758976560d7360e97c" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 42961 } } peers { permanent_uuid: "6e553496354446f1a839a2a86cc57c3e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36459 } } peers { permanent_uuid: "69dce9f1b04743e8ae9ceb0c185286b6" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45137 } }
I20260926 14:53:58.262522 31165 leader_election.cc:288] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [CANDIDATE]: Term 1 election: Requested vote from peers 6e553496354446f1a839a2a86cc57c3e (127.12.68.195:36459), 69dce9f1b04743e8ae9ceb0c185286b6 (127.12.68.194:45137)
I20260926 14:53:58.262820 31095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07b00f00e6024d488c7dcbf7546a7df5" candidate_uuid: "831f4b11260043758976560d7360e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e553496354446f1a839a2a86cc57c3e"
I20260926 14:53:58.262887 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07b00f00e6024d488c7dcbf7546a7df5" candidate_uuid: "831f4b11260043758976560d7360e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69dce9f1b04743e8ae9ceb0c185286b6"
I20260926 14:53:58.262936 31095 raft_consensus.cc:3037] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:58.262984 30961 raft_consensus.cc:3037] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:58.264133 30961 raft_consensus.cc:2454] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831f4b11260043758976560d7360e97c in term 1.
I20260926 14:53:58.264128 31095 raft_consensus.cc:2454] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831f4b11260043758976560d7360e97c in term 1.
I20260926 14:53:58.264392 30762 leader_election.cc:302] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69dce9f1b04743e8ae9ceb0c185286b6, 831f4b11260043758976560d7360e97c; no voters:
I20260926 14:53:58.264534 31165 raft_consensus.cc:2781] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:53:58.264717 31165 raft_consensus.cc:686] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [term 1 LEADER]: Becoming Leader. State: Replica: 831f4b11260043758976560d7360e97c, State: Running, Role: LEADER
I20260926 14:53:58.265007 31165 consensus_queue.cc:237] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831f4b11260043758976560d7360e97c" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 42961 } } peers { permanent_uuid: "6e553496354446f1a839a2a86cc57c3e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36459 } } peers { permanent_uuid: "69dce9f1b04743e8ae9ceb0c185286b6" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45137 } }
I20260926 14:53:58.267643 30680 catalog_manager.cc:5495] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c reported cstate change: term changed from 0 to 1, leader changed from <none> to 831f4b11260043758976560d7360e97c (127.12.68.193). New cstate: current_term: 1 leader_uuid: "831f4b11260043758976560d7360e97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831f4b11260043758976560d7360e97c" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 42961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e553496354446f1a839a2a86cc57c3e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69dce9f1b04743e8ae9ceb0c185286b6" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45137 } health_report { overall_health: UNKNOWN } } }
W20260926 14:53:58.271375 30874 tablet.cc:2369] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:53:58.279462 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.276203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.12.68.193:42961)
0926 14:53:58.276320 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:58.276324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:58.276336 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:53:58.276486 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:58.276488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:58.276536 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:58.276610 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:53:58.276860 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.277393 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.278218 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.278425 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.278524 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.278763 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:53:58.279121 (+ 358us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:53:58.279223 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:58.279283 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 14:53:58.279316 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:53:58.284828 30961 raft_consensus.cc:1261] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6 [term 1 FOLLOWER]: Refusing update from remote peer 831f4b11260043758976560d7360e97c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:53:58.284828 31095 raft_consensus.cc:1261] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e [term 1 FOLLOWER]: Refusing update from remote peer 831f4b11260043758976560d7360e97c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:53:58.285452 31168 consensus_queue.cc:1035] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "69dce9f1b04743e8ae9ceb0c185286b6" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:53:58.285634 31165 consensus_queue.cc:1035] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e553496354446f1a839a2a86cc57c3e" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 36459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:53:58.291770 31171 mvcc.cc:205] Tried to move back new op lower bound from 7333619459207802880 to 7333619459135950848. Current Snapshot: MvccSnapshot[applied={T|T < 7333619459207802880}]
I20260926 14:53:58.292866 31173 mvcc.cc:205] Tried to move back new op lower bound from 7333619459207802880 to 7333619459135950848. Current Snapshot: MvccSnapshot[applied={T|T < 7333619459207802880}]
I20260926 14:53:58.294081 31172 mvcc.cc:205] Tried to move back new op lower bound from 7333619459207802880 to 7333619459135950848. Current Snapshot: MvccSnapshot[applied={T|T < 7333619459207802880}]
I20260926 14:53:58.301455 30680 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:53:58.301604 30680 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:53:58.303822 30680 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66d0398bd2fe4cc4bd80db3a3894e41f: Sending DeleteTablet for 3 replicas of tablet 07b00f00e6024d488c7dcbf7546a7df5
I20260926 14:53:58.304481 30807 tablet_service.cc:1514] Processing DeleteTablet for tablet 07b00f00e6024d488c7dcbf7546a7df5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:58 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:34424
I20260926 14:53:58.304672 31075 tablet_service.cc:1514] Processing DeleteTablet for tablet 07b00f00e6024d488c7dcbf7546a7df5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:58 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:43310
I20260926 14:53:58.304821 30941 tablet_service.cc:1514] Processing DeleteTablet for tablet 07b00f00e6024d488c7dcbf7546a7df5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:58 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:33442
I20260926 14:53:58.305281 31183 tablet_replica.cc:330] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6: stopping tablet replica
I20260926 14:53:58.305292 31181 tablet_replica.cc:330] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c: stopping tablet replica
I20260926 14:53:58.305564 31181 raft_consensus.cc:2229] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:53:58.305567 31183 raft_consensus.cc:2229] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:53:58.305716 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 30744
I20260926 14:53:58.305866 31183 raft_consensus.cc:2258] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:58.305943 31181 raft_consensus.cc:2258] T 07b00f00e6024d488c7dcbf7546a7df5 P 831f4b11260043758976560d7360e97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:58.306362 31182 tablet_replica.cc:330] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e: stopping tablet replica
I20260926 14:53:58.306553 31182 raft_consensus.cc:2229] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:53:58.306756 31182 raft_consensus.cc:2258] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:58.307483 31183 ts_tablet_manager.cc:1905] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:58.309420 31182 ts_tablet_manager.cc:1905] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:58.310058 31183 ts_tablet_manager.cc:1918] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:53:58.310130 31183 log.cc:1196] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/07b00f00e6024d488c7dcbf7546a7df5
I20260926 14:53:58.310333 31183 ts_tablet_manager.cc:1939] T 07b00f00e6024d488c7dcbf7546a7df5 P 69dce9f1b04743e8ae9ceb0c185286b6: Deleting consensus metadata
I20260926 14:53:58.311276 30668 catalog_manager.cc:4841] TS 69dce9f1b04743e8ae9ceb0c185286b6 (127.12.68.194:45137): tablet 07b00f00e6024d488c7dcbf7546a7df5 (table test-table [id=d81f300ea7fa4ed78ced01a825405c2a]) successfully deleted
I20260926 14:53:58.311801 31182 ts_tablet_manager.cc:1918] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:53:58.311905 31182 log.cc:1196] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/07b00f00e6024d488c7dcbf7546a7df5
I20260926 14:53:58.312194 31182 ts_tablet_manager.cc:1939] T 07b00f00e6024d488c7dcbf7546a7df5 P 6e553496354446f1a839a2a86cc57c3e: Deleting consensus metadata
I20260926 14:53:58.313215 30668 catalog_manager.cc:4841] TS 6e553496354446f1a839a2a86cc57c3e (127.12.68.195:36459): tablet 07b00f00e6024d488c7dcbf7546a7df5 (table test-table [id=d81f300ea7fa4ed78ced01a825405c2a]) successfully deleted
I20260926 14:53:58.313592 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 30878
W20260926 14:53:58.313604 30667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:42961 (error 108)
I20260926 14:53:58.314878 31152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.314658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:42961 (local address 127.0.0.1:34450)
0926 14:53:58.314758 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:58.314811 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:42961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:53:58.315006 30667 catalog_manager.cc:4584] TS 831f4b11260043758976560d7360e97c (127.12.68.193:42961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07b00f00e6024d488c7dcbf7546a7df5: Network error: Client connection negotiation failed: client connection to 127.12.68.193:42961: connect: Connection refused (error 111)
I20260926 14:53:58.318917 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 31012
I20260926 14:53:58.323444 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 30650
2026-09-26T14:53:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2104 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T14:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:53:58Z Disabled control of system clock
I20260926 14:53:58.350497 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:45245
--webserver_interface=127.12.68.254
--webserver_port=0
--builtin_ntp_servers=127.12.68.212:33469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:45245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:53:58.453341 31188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:58.453567 31188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:58.453600 31188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:58.457584 31188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:53:58.457639 31188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:58.457653 31188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:58.457667 31188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:53:58.457682 31188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:53:58.462270 31188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:45245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:45245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:58.463279 31188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:58.464596 31188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:58.472244 31194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:58.472260 31193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:58.472386 31188 server_base.cc:1034] running on GCE node
W20260926 14:53:58.472260 31196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:58.473024 31188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:58.474044 31188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:58.475216 31188 hybrid_clock.cc:648] HybridClock initialized: now 1790434438475199 us; error 28 us; skew 500 ppm
I20260926 14:53:58.476905 31188 webserver.cc:460] Webserver started at http://127.12.68.254:41193/ using document root <none> and password file <none>
I20260926 14:53:58.477415 31188 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:58.477478 31188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:58.477662 31188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:58.479537 31188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "a3155e16d66b406fb0046e9d3f762414"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-24mn"
server_key: "d848ecadb8afd4bd771c8df86d76ac6e"
server_key_iv: "98d3e34f6f8ea7b54077b774abe183e5"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.480083 31188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "a3155e16d66b406fb0046e9d3f762414"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-24mn"
server_key: "d848ecadb8afd4bd771c8df86d76ac6e"
server_key_iv: "98d3e34f6f8ea7b54077b774abe183e5"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.483366 31188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:53:58.485579 31202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:58.486716 31188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:53:58.486821 31188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "a3155e16d66b406fb0046e9d3f762414"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-24mn"
server_key: "d848ecadb8afd4bd771c8df86d76ac6e"
server_key_iv: "98d3e34f6f8ea7b54077b774abe183e5"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.486907 31188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:58.511996 31188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:58.512651 31188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:58.512815 31188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:58.518939 31188 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:45245
I20260926 14:53:58.518945 31254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:45245 every 8 connection(s)
I20260926 14:53:58.519791 31188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:53:58.522681 31255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:58.525278 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 31188
I20260926 14:53:58.525389 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:53:58.525642 12563 external_mini_cluster.cc:1428] Setting key f262c6879285fe975d36a7d2475c8644
I20260926 14:53:58.528396 31255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414: Bootstrap starting.
I20260926 14:53:58.530894 31255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:58.531638 31255 log.cc:824] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:58.533623 31255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414: No bootstrap required, opened a new log
I20260926 14:53:58.535513 31258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.526919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.12.68.254:45245)
0926 14:53:58.527558 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:58.527582 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:58.527637 (+ 55us) server_negotiation.cc:407] Connection header received
0926 14:53:58.528437 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:58.528459 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:58.528934 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:58.529282 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:58.529755 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.531176 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.532080 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.532663 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.532801 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.533530 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:58.533535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:58.533551 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:53:58.534028 (+ 477us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:58.534093 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:53:58.534119 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:58.534211 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:58.534301 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 14:53:58.534624 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":97,"threads_started":1}
I20260926 14:53:58.537727 31255 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3155e16d66b406fb0046e9d3f762414" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45245 } }
I20260926 14:53:58.537959 31255 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:58.538002 31255 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3155e16d66b406fb0046e9d3f762414, State: Initialized, Role: FOLLOWER
I20260926 14:53:58.538340 31255 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3155e16d66b406fb0046e9d3f762414" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45245 } }
I20260926 14:53:58.538439 31255 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:53:58.538479 31255 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:53:58.538540 31255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:58.539444 31255 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3155e16d66b406fb0046e9d3f762414" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45245 } }
I20260926 14:53:58.539753 31255 leader_election.cc:302] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3155e16d66b406fb0046e9d3f762414; no voters:
I20260926 14:53:58.540020 31255 leader_election.cc:288] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:53:58.540133 31260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:53:58.540309 31260 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [term 1 LEADER]: Becoming Leader. State: Replica: a3155e16d66b406fb0046e9d3f762414, State: Running, Role: LEADER
I20260926 14:53:58.540601 31260 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3155e16d66b406fb0046e9d3f762414" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45245 } }
I20260926 14:53:58.541035 31255 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:53:58.541986 31262 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3155e16d66b406fb0046e9d3f762414. Latest consensus state: current_term: 1 leader_uuid: "a3155e16d66b406fb0046e9d3f762414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3155e16d66b406fb0046e9d3f762414" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45245 } } }
I20260926 14:53:58.542105 31262 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:58.542356 31261 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3155e16d66b406fb0046e9d3f762414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3155e16d66b406fb0046e9d3f762414" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45245 } } }
I20260926 14:53:58.542428 31269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:53:58.542445 31261 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:58.545555 31269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:53:58.550541 31269 catalog_manager.cc:1347] Generated new cluster ID: e0f4791615b0416ebd3c279f11fa209c
I20260926 14:53:58.550604 31269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:53:58.563650 31269 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:53:58.564481 31269 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:53:58.568178 31269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414: Generated new TSK 0
I20260926 14:53:58.568876 31269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:53:58.571355 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45245
--builtin_ntp_servers=127.12.68.212:33469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:53:58.676100 31279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:58.676333 31279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:58.676369 31279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:58.680294 31279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:58.680342 31279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:58.680394 31279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:53:58.684965 31279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:58.685936 31279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:58.687209 31279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:58.696043 31285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:58.696053 31284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:58.696203 31279 server_base.cc:1034] running on GCE node
W20260926 14:53:58.696082 31287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:58.696686 31279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:58.697245 31279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:58.698402 31279 hybrid_clock.cc:648] HybridClock initialized: now 1790434438698385 us; error 29 us; skew 500 ppm
I20260926 14:53:58.700294 31279 webserver.cc:460] Webserver started at http://127.12.68.193:39167/ using document root <none> and password file <none>
I20260926 14:53:58.700795 31279 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:58.700858 31279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:58.701040 31279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:58.702891 31279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "9c5baab2354c432b9a81f2d2b8867315"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-24mn"
server_key: "cc7cb4474612225628dfea3013595c96"
server_key_iv: "074102370b7b2a74237353d4562a8ff1"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.703393 31279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "9c5baab2354c432b9a81f2d2b8867315"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-24mn"
server_key: "cc7cb4474612225628dfea3013595c96"
server_key_iv: "074102370b7b2a74237353d4562a8ff1"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.706693 31279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:53:58.708989 31293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:58.710098 31279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:53:58.710189 31279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "9c5baab2354c432b9a81f2d2b8867315"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-24mn"
server_key: "cc7cb4474612225628dfea3013595c96"
server_key_iv: "074102370b7b2a74237353d4562a8ff1"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.710279 31279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:58.724782 31279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:58.725351 31279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:58.725494 31279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:58.725980 31279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:58.726800 31279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:58.726843 31279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:58.726881 31279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:58.726908 31279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:58.734581 31279 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:45657
I20260926 14:53:58.734616 31406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:45657 every 8 connection(s)
I20260926 14:53:58.735373 31279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:53:58.736619 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 31279
I20260926 14:53:58.736729 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:53:58.737021 12563 external_mini_cluster.cc:1428] Setting key e6569e6d6c38087c02f5c01a397376bc
I20260926 14:53:58.738999 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45245
--builtin_ntp_servers=127.12.68.212:33469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:53:58.745173 31258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.737008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:37113 (local address 127.12.68.254:45245)
0926 14:53:58.737143 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:58.737148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:58.738174 (+ 1026us) server_negotiation.cc:407] Connection header received
0926 14:53:58.739804 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:58.739806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:58.739850 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:58.739975 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:58.740907 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.741657 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.743243 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.743456 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.743534 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.744755 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:58.744758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:58.744760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:53:58.744771 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:58.744790 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:53:58.744797 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:58.744852 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:58.745015 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 14:53:58.745064 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:53:58.746361 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.737429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45245 (local address 127.12.68.193:37113)
0926 14:53:58.738049 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:58.738086 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:58.739597 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:58.740136 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:58.740144 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:58.740243 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:58.740779 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.740791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.741781 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.741785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.743136 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.743148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.743589 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.743594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.743784 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.744473 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:53:58.744475 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:58.744529 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:53:58.744612 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:58.744880 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:58.744888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:58.744894 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:53:58.745019 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 14:53:58.745406 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260926 14:53:58.747432 31407 heartbeater.cc:347] Connected to a master server at 127.12.68.254:45245
I20260926 14:53:58.747637 31407 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:58.748121 31407 heartbeater.cc:511] Master 127.12.68.254:45245 requested a full tablet report, sending...
I20260926 14:53:58.749418 31219 ts_manager.cc:194] Registered new tserver with Master: 9c5baab2354c432b9a81f2d2b8867315 (127.12.68.193:45657)
I20260926 14:53:58.750478 31219 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.68.193:37113
W20260926 14:53:58.846671 31410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:58.846892 31410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:58.846931 31410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:58.850823 31410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:58.850869 31410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:58.850924 31410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:53:58.855472 31410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:58.856442 31410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:58.857772 31410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:58.866652 31416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:58.866614 31415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:58.866633 31418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:58.866745 31410 server_base.cc:1034] running on GCE node
I20260926 14:53:58.867261 31410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:58.867807 31410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:58.868989 31410 hybrid_clock.cc:648] HybridClock initialized: now 1790434438868971 us; error 30 us; skew 500 ppm
I20260926 14:53:58.870659 31410 webserver.cc:460] Webserver started at http://127.12.68.194:36591/ using document root <none> and password file <none>
I20260926 14:53:58.871199 31410 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:58.871263 31410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:58.871454 31410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:58.873385 31410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "a1675be26655431f9d252674906856f8"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-24mn"
server_key: "263046063f271b90f0aedfcc297dfa79"
server_key_iv: "a993f235cfe1ec2d26e1e1af642f48f9"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.873870 31410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "a1675be26655431f9d252674906856f8"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-24mn"
server_key: "263046063f271b90f0aedfcc297dfa79"
server_key_iv: "a993f235cfe1ec2d26e1e1af642f48f9"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.877236 31410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:53:58.879390 31424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:58.880509 31410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:53:58.880615 31410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "a1675be26655431f9d252674906856f8"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-24mn"
server_key: "263046063f271b90f0aedfcc297dfa79"
server_key_iv: "a993f235cfe1ec2d26e1e1af642f48f9"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.880702 31410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:58.894760 31410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:58.895314 31410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:58.895463 31410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:58.896008 31410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:58.896840 31410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:58.896884 31410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:58.896932 31410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:58.896961 31410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:58.904572 31410 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:36853
I20260926 14:53:58.904624 31537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:36853 every 8 connection(s)
I20260926 14:53:58.905350 31410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:53:58.913688 31258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.906957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:54159 (local address 127.12.68.254:45245)
0926 14:53:58.907118 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:58.907123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:58.908178 (+ 1055us) server_negotiation.cc:407] Connection header received
0926 14:53:58.909487 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:58.909490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:58.909537 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:58.909609 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:58.910439 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.910974 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.912080 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.912302 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.912380 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.913275 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:58.913279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:58.913280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:53:58.913292 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:58.913311 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:53:58.913319 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:58.913376 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:58.913472 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 14:53:58.913528 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 14:53:58.914414 31540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.907368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45245 (local address 127.12.68.194:54159)
0926 14:53:58.908019 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:58.908057 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:58.909271 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:58.909767 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:58.909773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:58.909854 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:58.910316 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.910328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.911059 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.911061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.911962 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:58.911972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.912410 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.912412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:58.912540 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.913022 (+ 482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:53:58.913024 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:58.913061 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:53:58.913142 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:58.913380 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:58.913384 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:58.913388 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:53:58.913467 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 14:53:58.913687 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":99,"threads_started":1}
I20260926 14:53:58.915220 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 31410
I20260926 14:53:58.915339 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:53:58.915503 31538 heartbeater.cc:347] Connected to a master server at 127.12.68.254:45245
I20260926 14:53:58.915656 12563 external_mini_cluster.cc:1428] Setting key 0c1a6c2c150d31bada84f5e60357d053
I20260926 14:53:58.915723 31538 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:58.916246 31538 heartbeater.cc:511] Master 127.12.68.254:45245 requested a full tablet report, sending...
I20260926 14:53:58.917171 31219 ts_manager.cc:194] Registered new tserver with Master: a1675be26655431f9d252674906856f8 (127.12.68.194:36853)
I20260926 14:53:58.917601 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45245
--builtin_ntp_servers=127.12.68.212:33469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:53:58.917872 31219 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.68.194:54159
W20260926 14:53:59.022033 31541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:59.022279 31541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:59.022330 31541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:59.026254 31541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:59.026304 31541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:59.026376 31541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:53:59.030992 31541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:53:59.031977 31541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:59.033253 31541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:59.041975 31547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:59.042026 31546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:59.042207 31541 server_base.cc:1034] running on GCE node
W20260926 14:53:59.042039 31549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:59.042659 31541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:59.043246 31541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:59.044389 31541 hybrid_clock.cc:648] HybridClock initialized: now 1790434439044369 us; error 45 us; skew 500 ppm
I20260926 14:53:59.046059 31541 webserver.cc:460] Webserver started at http://127.12.68.195:44547/ using document root <none> and password file <none>
I20260926 14:53:59.046569 31541 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:59.046633 31541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:59.046816 31541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:59.048763 31541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "be5f4b7754094ac1a8dbb9f239b3d427"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-24mn"
server_key: "e9bc72c51923fadfa3074ea362790678"
server_key_iv: "c3eb0a563d61e383f9e45574f1e44fd2"
server_key_version: "encryptionkey@0"
I20260926 14:53:59.049266 31541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "be5f4b7754094ac1a8dbb9f239b3d427"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-24mn"
server_key: "e9bc72c51923fadfa3074ea362790678"
server_key_iv: "c3eb0a563d61e383f9e45574f1e44fd2"
server_key_version: "encryptionkey@0"
I20260926 14:53:59.052639 31541 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:53:59.054870 31555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:59.056071 31541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:53:59.056173 31541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "be5f4b7754094ac1a8dbb9f239b3d427"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-24mn"
server_key: "e9bc72c51923fadfa3074ea362790678"
server_key_iv: "c3eb0a563d61e383f9e45574f1e44fd2"
server_key_version: "encryptionkey@0"
I20260926 14:53:59.056260 31541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:59.069482 31541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:59.070042 31541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:59.070186 31541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:59.070672 31541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:59.071507 31541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:59.071555 31541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:59.071604 31541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:59.071630 31541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:59.079345 31541 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:42235
I20260926 14:53:59.079371 31668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:42235 every 8 connection(s)
I20260926 14:53:59.080159 31541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:53:59.083112 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 31541
I20260926 14:53:59.083196 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:53:59.083426 12563 external_mini_cluster.cc:1428] Setting key c39658ef3309d0f5892d648948532c52
I20260926 14:53:59.088614 31258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.081683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:48785 (local address 127.12.68.254:45245)
0926 14:53:59.081791 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:59.081795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:59.082726 (+ 931us) server_negotiation.cc:407] Connection header received
0926 14:53:59.084199 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:59.084203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:59.084254 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:59.084328 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:59.085505 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.086023 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.087083 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.087290 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.087357 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.088287 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:59.088290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:59.088292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:53:59.088301 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:59.088317 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:53:59.088323 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:59.088369 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:59.088459 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 14:53:59.088501 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:53:59.089346 31671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.082011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45245 (local address 127.12.68.195:48785)
0926 14:53:59.082593 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:59.082628 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:59.083955 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:59.084662 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:59.084669 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:59.084767 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:59.085391 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.085405 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.086125 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.086128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.086976 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.086984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.087390 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.087392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.087518 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.088089 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:53:59.088090 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:59.088127 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:53:59.088182 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:59.088373 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:59.088377 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:59.088381 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:53:59.088454 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:53:59.088648 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":100,"threads_started":1}
I20260926 14:53:59.090301 31669 heartbeater.cc:347] Connected to a master server at 127.12.68.254:45245
I20260926 14:53:59.090507 31669 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:59.090965 31669 heartbeater.cc:511] Master 127.12.68.254:45245 requested a full tablet report, sending...
I20260926 14:53:59.091938 31219 ts_manager.cc:194] Registered new tserver with Master: be5f4b7754094ac1a8dbb9f239b3d427 (127.12.68.195:42235)
I20260926 14:53:59.092479 31219 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.68.195:48785
I20260926 14:53:59.096892 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:53:59.102196 31258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.098626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.12.68.254:45245)
0926 14:53:59.098737 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:59.098741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:59.098907 (+ 166us) server_negotiation.cc:407] Connection header received
0926 14:53:59.099016 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:59.099019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:59.099061 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:59.099123 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:59.099379 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.099870 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.100706 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.100957 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.101041 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.101762 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:59.101765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:59.101768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:53:59.101779 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:59.101797 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:53:59.101805 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:59.101903 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:59.102034 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 14:53:59.102069 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:53:59.104486 31219 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:53:59.107319 31219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:53:59.120700 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.115663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.12.68.194:36853)
0926 14:53:59.116445 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:59.116456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:59.116476 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:53:59.116553 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:59.116557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:59.116807 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:59.117022 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:59.117146 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.118256 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.119496 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.120050 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.120142 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.120324 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:59.120445 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 14:53:59.120533 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":94,"threads_started":1}
I20260926 14:53:59.120761 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.115046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:45657 (local address 127.0.0.1:49340)
0926 14:53:59.115731 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:59.115778 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:59.115919 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:59.116804 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:59.116807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:59.116825 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:59.117023 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.117030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.117988 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.117991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.119281 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.119292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.120344 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.120348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.120435 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.120459 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:53:59.120506 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 14:53:59.120593 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":16,"spinlock_wait_cycles":1792,"thread_start_us":94,"threads_started":1}
I20260926 14:53:59.120991 31683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.114954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:42235 (local address 127.0.0.1:37770)
0926 14:53:59.115733 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:59.115780 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:59.115932 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:59.116649 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:59.116655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:59.116677 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:59.116809 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.116816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.118364 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.118368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.119678 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.119687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.120672 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.120675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.120762 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.120785 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:53:59.120846 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:53:59.120901 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":149,"spinlock_wait_cycles":107264,"thread_start_us":65,"threads_started":1}
I20260926 14:53:59.121345 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.115089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.12.68.195:42235)
0926 14:53:59.116189 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:59.116201 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:59.116223 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:53:59.116290 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:59.116296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:59.116487 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:59.116957 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:59.117094 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.118256 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.119801 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.120581 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.120671 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.121067 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:59.121116 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 14:53:59.121199 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1003,"thread_start_us":89,"threads_started":1}
I20260926 14:53:59.121361 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.116018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.12.68.193:45657)
0926 14:53:59.116308 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:59.116319 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:59.116342 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:53:59.116414 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:59.116419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:59.116613 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:59.116828 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:59.117148 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.117875 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.119407 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.120216 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.120920 (+ 704us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.121104 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:59.121145 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 14:53:59.121221 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
I20260926 14:53:59.121599 31681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.115272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:36853 (local address 127.0.0.1:43260)
0926 14:53:59.115731 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:59.115778 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:59.115919 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:59.116984 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:59.116987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:59.117001 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:59.117075 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.117080 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.118398 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.118401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.119401 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.119411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.120157 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.120161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.120276 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.120311 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 14:53:59.120411 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 14:53:59.120533 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":67,"spinlock_wait_cycles":1408,"thread_start_us":95,"threads_started":1}
I20260926 14:53:59.124127 31603 tablet_service.cc:1467] Processing CreateTablet for tablet b5d3fd3be454403fb2ebd24227b3044b (DEFAULT_TABLE table=test-table [id=40975401bf494270b63e56538ffc4292]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:59.124205 31472 tablet_service.cc:1467] Processing CreateTablet for tablet b5d3fd3be454403fb2ebd24227b3044b (DEFAULT_TABLE table=test-table [id=40975401bf494270b63e56538ffc4292]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:59.124289 31341 tablet_service.cc:1467] Processing CreateTablet for tablet b5d3fd3be454403fb2ebd24227b3044b (DEFAULT_TABLE table=test-table [id=40975401bf494270b63e56538ffc4292]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:59.125209 31472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d3fd3be454403fb2ebd24227b3044b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:59.125211 31603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d3fd3be454403fb2ebd24227b3044b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:59.125385 31341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d3fd3be454403fb2ebd24227b3044b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:59.131939 31686 tablet_bootstrap.cc:492] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427: Bootstrap starting.
I20260926 14:53:59.132555 31688 tablet_bootstrap.cc:492] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315: Bootstrap starting.
I20260926 14:53:59.132937 31687 tablet_bootstrap.cc:492] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8: Bootstrap starting.
I20260926 14:53:59.133873 31686 tablet_bootstrap.cc:654] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:59.134536 31686 log.cc:824] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:59.134594 31688 tablet_bootstrap.cc:654] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:59.134717 31687 tablet_bootstrap.cc:654] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:59.135372 31688 log.cc:824] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:59.135372 31687 log.cc:824] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:59.136291 31686 tablet_bootstrap.cc:492] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427: No bootstrap required, opened a new log
I20260926 14:53:59.136466 31686 ts_tablet_manager.cc:1397] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:53:59.137198 31688 tablet_bootstrap.cc:492] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315: No bootstrap required, opened a new log
I20260926 14:53:59.137198 31687 tablet_bootstrap.cc:492] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8: No bootstrap required, opened a new log
I20260926 14:53:59.137365 31688 ts_tablet_manager.cc:1397] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:53:59.137365 31687 ts_tablet_manager.cc:1397] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:53:59.140040 31686 raft_consensus.cc:348] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be5f4b7754094ac1a8dbb9f239b3d427" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 42235 } } peers { permanent_uuid: "9c5baab2354c432b9a81f2d2b8867315" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 45657 } } peers { permanent_uuid: "a1675be26655431f9d252674906856f8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36853 } }
I20260926 14:53:59.140242 31686 raft_consensus.cc:374] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:59.140292 31686 raft_consensus.cc:729] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be5f4b7754094ac1a8dbb9f239b3d427, State: Initialized, Role: FOLLOWER
I20260926 14:53:59.140688 31686 consensus_queue.cc:260] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be5f4b7754094ac1a8dbb9f239b3d427" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 42235 } } peers { permanent_uuid: "9c5baab2354c432b9a81f2d2b8867315" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 45657 } } peers { permanent_uuid: "a1675be26655431f9d252674906856f8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36853 } }
I20260926 14:53:59.141520 31686 ts_tablet_manager.cc:1428] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:53:59.141378 31687 raft_consensus.cc:348] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be5f4b7754094ac1a8dbb9f239b3d427" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 42235 } } peers { permanent_uuid: "9c5baab2354c432b9a81f2d2b8867315" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 45657 } } peers { permanent_uuid: "a1675be26655431f9d252674906856f8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36853 } }
I20260926 14:53:59.141376 31688 raft_consensus.cc:348] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be5f4b7754094ac1a8dbb9f239b3d427" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 42235 } } peers { permanent_uuid: "9c5baab2354c432b9a81f2d2b8867315" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 45657 } } peers { permanent_uuid: "a1675be26655431f9d252674906856f8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36853 } }
I20260926 14:53:59.141600 31687 raft_consensus.cc:374] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:59.141600 31688 raft_consensus.cc:374] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:59.141657 31688 raft_consensus.cc:729] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5baab2354c432b9a81f2d2b8867315, State: Initialized, Role: FOLLOWER
I20260926 14:53:59.141690 31687 raft_consensus.cc:729] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1675be26655431f9d252674906856f8, State: Initialized, Role: FOLLOWER
I20260926 14:53:59.141850 31669 heartbeater.cc:503] Master 127.12.68.254:45245 was elected leader, sending a full tablet report...
I20260926 14:53:59.142079 31688 consensus_queue.cc:260] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be5f4b7754094ac1a8dbb9f239b3d427" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 42235 } } peers { permanent_uuid: "9c5baab2354c432b9a81f2d2b8867315" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 45657 } } peers { permanent_uuid: "a1675be26655431f9d252674906856f8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36853 } }
I20260926 14:53:59.142081 31687 consensus_queue.cc:260] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be5f4b7754094ac1a8dbb9f239b3d427" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 42235 } } peers { permanent_uuid: "9c5baab2354c432b9a81f2d2b8867315" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 45657 } } peers { permanent_uuid: "a1675be26655431f9d252674906856f8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36853 } }
I20260926 14:53:59.142869 31407 heartbeater.cc:503] Master 127.12.68.254:45245 was elected leader, sending a full tablet report...
I20260926 14:53:59.142974 31688 ts_tablet_manager.cc:1428] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:53:59.142998 31687 ts_tablet_manager.cc:1428] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260926 14:53:59.143812 31538 heartbeater.cc:503] Master 127.12.68.254:45245 was elected leader, sending a full tablet report...
W20260926 14:53:59.156121 31539 tablet.cc:2369] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:53:59.210495 31694 raft_consensus.cc:482] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:53:59.210686 31694 raft_consensus.cc:504] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be5f4b7754094ac1a8dbb9f239b3d427" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 42235 } } peers { permanent_uuid: "9c5baab2354c432b9a81f2d2b8867315" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 45657 } } peers { permanent_uuid: "a1675be26655431f9d252674906856f8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36853 } }
I20260926 14:53:59.211771 31694 leader_election.cc:288] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be5f4b7754094ac1a8dbb9f239b3d427 (127.12.68.195:42235), 9c5baab2354c432b9a81f2d2b8867315 (127.12.68.193:45657)
I20260926 14:53:59.215417 31540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.212018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:45657 (local address 127.12.68.194:47109)
0926 14:53:59.212186 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:59.212201 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:59.212284 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:59.212543 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:59.212546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:59.212565 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:59.212679 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.212686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.213540 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.213544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.214376 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.214386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.215012 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.215016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.215142 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.215164 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:53:59.215231 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:53:59.215290 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 14:53:59.215493 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.212130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:36819 (local address 127.12.68.195:42235)
0926 14:53:59.212237 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:59.212242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:59.212522 (+ 280us) server_negotiation.cc:407] Connection header received
0926 14:53:59.212630 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:59.212633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:59.212681 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:59.212755 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:59.212993 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.213540 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.214591 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.215008 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.215117 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.215162 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:59.215324 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 14:53:59.215391 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:53:59.215489 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.212091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:47109 (local address 127.12.68.193:45657)
0926 14:53:59.212225 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:59.212229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:59.212245 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:53:59.212391 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:59.212394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:59.212448 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:59.212525 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:59.212777 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.213381 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.214475 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.214904 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.214993 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.215031 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:59.215255 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 14:53:59.215374 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 14:53:59.215723 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.212065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:42235 (local address 127.12.68.194:36819)
0926 14:53:59.212425 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:59.212438 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:59.212536 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:59.212770 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:59.212772 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:59.212788 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:59.212901 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.212907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.213646 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.213648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.214475 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.214484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.215138 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.215141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.215215 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.215237 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:53:59.215310 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:53:59.215372 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":113,"threads_started":1}
I20260926 14:53:59.216168 31361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d3fd3be454403fb2ebd24227b3044b" candidate_uuid: "a1675be26655431f9d252674906856f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5baab2354c432b9a81f2d2b8867315" is_pre_election: true
I20260926 14:53:59.216460 31361 raft_consensus.cc:2454] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1675be26655431f9d252674906856f8 in term 0.
I20260926 14:53:59.216476 31623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d3fd3be454403fb2ebd24227b3044b" candidate_uuid: "a1675be26655431f9d252674906856f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be5f4b7754094ac1a8dbb9f239b3d427" is_pre_election: true
I20260926 14:53:59.216727 31623 raft_consensus.cc:2454] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1675be26655431f9d252674906856f8 in term 0.
I20260926 14:53:59.216910 31428 leader_election.cc:302] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c5baab2354c432b9a81f2d2b8867315, a1675be26655431f9d252674906856f8; no voters:
I20260926 14:53:59.217206 31694 raft_consensus.cc:2781] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:53:59.217279 31694 raft_consensus.cc:482] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:53:59.217330 31694 raft_consensus.cc:3037] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:59.218328 31694 raft_consensus.cc:504] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be5f4b7754094ac1a8dbb9f239b3d427" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 42235 } } peers { permanent_uuid: "9c5baab2354c432b9a81f2d2b8867315" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 45657 } } peers { permanent_uuid: "a1675be26655431f9d252674906856f8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36853 } }
I20260926 14:53:59.218645 31694 leader_election.cc:288] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [CANDIDATE]: Term 1 election: Requested vote from peers be5f4b7754094ac1a8dbb9f239b3d427 (127.12.68.195:42235), 9c5baab2354c432b9a81f2d2b8867315 (127.12.68.193:45657)
I20260926 14:53:59.218945 31623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d3fd3be454403fb2ebd24227b3044b" candidate_uuid: "a1675be26655431f9d252674906856f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be5f4b7754094ac1a8dbb9f239b3d427"
I20260926 14:53:59.218962 31361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d3fd3be454403fb2ebd24227b3044b" candidate_uuid: "a1675be26655431f9d252674906856f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5baab2354c432b9a81f2d2b8867315"
I20260926 14:53:59.219063 31623 raft_consensus.cc:3037] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:59.219062 31361 raft_consensus.cc:3037] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:59.220269 31361 raft_consensus.cc:2454] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1675be26655431f9d252674906856f8 in term 1.
I20260926 14:53:59.220269 31623 raft_consensus.cc:2454] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1675be26655431f9d252674906856f8 in term 1.
I20260926 14:53:59.220562 31425 leader_election.cc:302] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1675be26655431f9d252674906856f8, be5f4b7754094ac1a8dbb9f239b3d427; no voters:
I20260926 14:53:59.220784 31694 raft_consensus.cc:2781] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:53:59.220971 31694 raft_consensus.cc:686] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [term 1 LEADER]: Becoming Leader. State: Replica: a1675be26655431f9d252674906856f8, State: Running, Role: LEADER
I20260926 14:53:59.221251 31694 consensus_queue.cc:237] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be5f4b7754094ac1a8dbb9f239b3d427" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 42235 } } peers { permanent_uuid: "9c5baab2354c432b9a81f2d2b8867315" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 45657 } } peers { permanent_uuid: "a1675be26655431f9d252674906856f8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36853 } }
I20260926 14:53:59.224087 31219 catalog_manager.cc:5495] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1675be26655431f9d252674906856f8 (127.12.68.194). New cstate: current_term: 1 leader_uuid: "a1675be26655431f9d252674906856f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be5f4b7754094ac1a8dbb9f239b3d427" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 42235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c5baab2354c432b9a81f2d2b8867315" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 45657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1675be26655431f9d252674906856f8" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 36853 } health_report { overall_health: HEALTHY } } }
W20260926 14:53:59.236364 31408 tablet.cc:2369] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:53:59.241369 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.237996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43274 (local address 127.12.68.194:36853)
0926 14:53:59.238133 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:59.238137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:59.238152 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:53:59.238222 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:59.238224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:59.238288 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:59.238375 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:53:59.238610 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.239095 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.239932 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.240160 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.240246 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.240567 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:53:59.240939 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:53:59.241062 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:59.241180 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 14:53:59.241213 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 14:53:59.246687 31361 raft_consensus.cc:1261] T b5d3fd3be454403fb2ebd24227b3044b P 9c5baab2354c432b9a81f2d2b8867315 [term 1 FOLLOWER]: Refusing update from remote peer a1675be26655431f9d252674906856f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:53:59.246688 31623 raft_consensus.cc:1261] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427 [term 1 FOLLOWER]: Refusing update from remote peer a1675be26655431f9d252674906856f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:53:59.247337 31696 consensus_queue.cc:1035] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be5f4b7754094ac1a8dbb9f239b3d427" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:53:59.247502 31694 consensus_queue.cc:1035] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c5baab2354c432b9a81f2d2b8867315" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 45657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:53:59.254508 31700 mvcc.cc:205] Tried to move back new op lower bound from 7333619463148105728 to 7333619463052451840. Current Snapshot: MvccSnapshot[applied={T|T < 7333619463148105728}]
I20260926 14:53:59.264865 31219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:53:59.265025 31219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:53:59.267181 31219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3155e16d66b406fb0046e9d3f762414: Sending DeleteTablet for 3 replicas of tablet b5d3fd3be454403fb2ebd24227b3044b
I20260926 14:53:59.267855 31603 tablet_service.cc:1514] Processing DeleteTablet for tablet b5d3fd3be454403fb2ebd24227b3044b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:59 UTC) from {username='slave'} at 127.0.0.1:37770
I20260926 14:53:59.268006 31472 tablet_service.cc:1514] Processing DeleteTablet for tablet b5d3fd3be454403fb2ebd24227b3044b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:59 UTC) from {username='slave'} at 127.0.0.1:43260
I20260926 14:53:59.268317 31341 tablet_service.cc:1514] Processing DeleteTablet for tablet b5d3fd3be454403fb2ebd24227b3044b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:59 UTC) from {username='slave'} at 127.0.0.1:49340
I20260926 14:53:59.268599 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 31279
I20260926 14:53:59.269515 31709 tablet_replica.cc:330] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427: stopping tablet replica
I20260926 14:53:59.269778 31709 raft_consensus.cc:2229] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:53:59.269987 31709 raft_consensus.cc:2258] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:59.270243 31710 tablet_replica.cc:330] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8: stopping tablet replica
I20260926 14:53:59.270430 31710 raft_consensus.cc:2229] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:53:59.270691 31710 raft_consensus.cc:2258] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:59.271940 31709 ts_tablet_manager.cc:1905] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:59.272316 31710 ts_tablet_manager.cc:1905] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:59.274580 31709 ts_tablet_manager.cc:1918] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:53:59.274676 31709 log.cc:1196] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/b5d3fd3be454403fb2ebd24227b3044b
I20260926 14:53:59.274942 31709 ts_tablet_manager.cc:1939] T b5d3fd3be454403fb2ebd24227b3044b P be5f4b7754094ac1a8dbb9f239b3d427: Deleting consensus metadata
I20260926 14:53:59.275815 31710 ts_tablet_manager.cc:1918] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:53:59.276063 31710 log.cc:1196] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/b5d3fd3be454403fb2ebd24227b3044b
I20260926 14:53:59.276078 31203 catalog_manager.cc:4841] TS be5f4b7754094ac1a8dbb9f239b3d427 (127.12.68.195:42235): tablet b5d3fd3be454403fb2ebd24227b3044b (table test-table [id=40975401bf494270b63e56538ffc4292]) successfully deleted
I20260926 14:53:59.276343 31710 ts_tablet_manager.cc:1939] T b5d3fd3be454403fb2ebd24227b3044b P a1675be26655431f9d252674906856f8: Deleting consensus metadata
W20260926 14:53:59.276912 31206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:45657 (error 108)
I20260926 14:53:59.277107 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 31410
I20260926 14:53:59.278440 31681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.278016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:45657 (local address 127.0.0.1:49352)
0926 14:53:59.278321 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:59.278374 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238}
W20260926 14:53:59.278563 31206 catalog_manager.cc:4584] TS 9c5baab2354c432b9a81f2d2b8867315 (127.12.68.193:45657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5d3fd3be454403fb2ebd24227b3044b: Network error: Client connection negotiation failed: client connection to 127.12.68.193:45657: connect: Connection refused (error 111)
I20260926 14:53:59.281852 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 31541
I20260926 14:53:59.282771 31681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.282009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:36853 (local address 127.0.0.1:43282)
0926 14:53:59.282669 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:59.282715 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.194:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592}
I20260926 14:53:59.286276 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 31188
2026-09-26T14:53:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (961 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:53:59 dist-test-slave-24mn krb5kdc[31715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:53:59 dist-test-slave-24mn krb5kdc[31715](info): set up 2 sockets
Sep 26 14:53:59 dist-test-slave-24mn krb5kdc[31715](info): commencing operation
krb5kdc: starting...
W20260926 14:54:00.227671 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:00 dist-test-slave-24mn krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:54:00.336544 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:42101
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:42101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:00.439002 31728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:00.439237 31728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:00.439275 31728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:00.443384 31728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:00.443434 31728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:00.443452 31728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:00.443467 31728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:00.443480 31728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:00.448488 31728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:42101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:42101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:00.449368 31728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:00.450253 31728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:00.457984 31733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:00.458036 31736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:00.457970 31734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:00.458480 31728 server_base.cc:1034] running on GCE node
I20260926 14:54:00.459120 31728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:00.460156 31728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:00.461303 31728 hybrid_clock.cc:648] HybridClock initialized: now 1790434440461279 us; error 38 us; skew 500 ppm
Sep 26 14:54:00 dist-test-slave-24mn krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:00.463757 31728 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:54:00.464547 31728 webserver.cc:460] Webserver started at http://127.12.68.254:44153/ using document root <none> and password file <none>
I20260926 14:54:00.465018 31728 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:00.465068 31728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:00.465221 31728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:00.467065 31728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "37dd2e69157848fe9da0f6b304f8f863"
format_stamp: "Formatted at 2026-09-26 14:54:00 on dist-test-slave-24mn"
server_key: "f8bd12a3d9b4bb02fb03e7aaf1951b21"
server_key_iv: "aa879c4f4c71ec6251d2cdf22b23caf9"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.467548 31728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "37dd2e69157848fe9da0f6b304f8f863"
format_stamp: "Formatted at 2026-09-26 14:54:00 on dist-test-slave-24mn"
server_key: "f8bd12a3d9b4bb02fb03e7aaf1951b21"
server_key_iv: "aa879c4f4c71ec6251d2cdf22b23caf9"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.470906 31728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 14:54:00.473112 31743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.474246 31728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:54:00.474347 31728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "37dd2e69157848fe9da0f6b304f8f863"
format_stamp: "Formatted at 2026-09-26 14:54:00 on dist-test-slave-24mn"
server_key: "f8bd12a3d9b4bb02fb03e7aaf1951b21"
server_key_iv: "aa879c4f4c71ec6251d2cdf22b23caf9"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.474443 31728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:00.489142 31728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:00.492075 31728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:00.492236 31728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:00.498610 31728 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:42101
I20260926 14:54:00.498631 31795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:42101 every 8 connection(s)
I20260926 14:54:00.499439 31728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:00.501180 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 31728
I20260926 14:54:00.501293 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:00.501567 12563 external_mini_cluster.cc:1428] Setting key d2973889f39e9128d129cd80dbbf310b
I20260926 14:54:00.502076 31796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:00.508455 31796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863: Bootstrap starting.
Sep 26 14:54:00 dist-test-slave-24mn krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:54:00.510794 31796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:00.511574 31796 log.cc:824] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:00.513257 31796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863: No bootstrap required, opened a new log
I20260926 14:54:00.514756 31799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.502973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54834 (local address 127.12.68.254:42101)
0926 14:54:00.503360 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.503374 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.503407 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:54:00.504084 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.504102 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.504410 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.504671 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:00.505465 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.506416 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.507559 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.508072 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.508198 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.510657 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:00.510662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:00.510673 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:00.510699 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:00.512287 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.512663 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.512668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.512672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.512773 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.513016 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.513019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.513021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.513398 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:00.513565 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.513843 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.514094 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
I20260926 14:54:00.516582 31796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37dd2e69157848fe9da0f6b304f8f863" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42101 } }
I20260926 14:54:00.516826 31796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:00.516883 31796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dd2e69157848fe9da0f6b304f8f863, State: Initialized, Role: FOLLOWER
I20260926 14:54:00.517263 31796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37dd2e69157848fe9da0f6b304f8f863" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42101 } }
I20260926 14:54:00.517385 31796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:00.517445 31796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:00.517534 31796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:00.518339 31796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37dd2e69157848fe9da0f6b304f8f863" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42101 } }
I20260926 14:54:00.518631 31796 leader_election.cc:302] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37dd2e69157848fe9da0f6b304f8f863; no voters:
I20260926 14:54:00.518889 31796 leader_election.cc:288] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:00.518972 31801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:00.519178 31801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [term 1 LEADER]: Becoming Leader. State: Replica: 37dd2e69157848fe9da0f6b304f8f863, State: Running, Role: LEADER
I20260926 14:54:00.519479 31801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37dd2e69157848fe9da0f6b304f8f863" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42101 } }
I20260926 14:54:00.519958 31796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:00.520923 31803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37dd2e69157848fe9da0f6b304f8f863. Latest consensus state: current_term: 1 leader_uuid: "37dd2e69157848fe9da0f6b304f8f863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37dd2e69157848fe9da0f6b304f8f863" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42101 } } }
I20260926 14:54:00.521035 31803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:00.521386 31810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:00.521378 31802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37dd2e69157848fe9da0f6b304f8f863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37dd2e69157848fe9da0f6b304f8f863" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 42101 } } }
I20260926 14:54:00.521920 31802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:00.524647 31810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:00.529865 31810 catalog_manager.cc:1347] Generated new cluster ID: f0636715696d4bd9a8e68994bf647a33
I20260926 14:54:00.529928 31810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:00.544723 31810 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:00.545570 31810 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:00.550354 31810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863: Generated new TSK 0
I20260926 14:54:00.550945 31810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
Entry for principal kudu/127.12.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab.
I20260926 14:54:00.578811 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42101
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:00.683100 31822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:00.683442 31822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:00.683531 31822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:00.687497 31822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:00.687551 31822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:00.687629 31822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:54:00.692734 31822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.193.keytab
--principal=kudu/127.12.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:00.693643 31822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:00.694574 31822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:00.703402 31828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:00.703437 31827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:00.703404 31830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:00.703727 31822 server_base.cc:1034] running on GCE node
I20260926 14:54:00.704106 31822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:00.704686 31822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:00.705855 31822 hybrid_clock.cc:648] HybridClock initialized: now 1790434440705838 us; error 26 us; skew 500 ppm
Sep 26 14:54:00 dist-test-slave-24mn krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:00.708426 31822 init.cc:377] Logged in from keytab as kudu/127.12.68.193@KRBTEST.COM (short username kudu)
I20260926 14:54:00.709214 31822 webserver.cc:460] Webserver started at http://127.12.68.193:35397/ using document root <none> and password file <none>
I20260926 14:54:00.709713 31822 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:00.709774 31822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:00.709949 31822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:00.711876 31822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "3ae2a2095d42461cbad197de15847cd1"
format_stamp: "Formatted at 2026-09-26 14:54:00 on dist-test-slave-24mn"
server_key: "df69be153cdb6b246c9c6b8d7b19c10d"
server_key_iv: "d2e1525f75f0b957c29e18dc1caafb97"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.712411 31822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "3ae2a2095d42461cbad197de15847cd1"
format_stamp: "Formatted at 2026-09-26 14:54:00 on dist-test-slave-24mn"
server_key: "df69be153cdb6b246c9c6b8d7b19c10d"
server_key_iv: "d2e1525f75f0b957c29e18dc1caafb97"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.715792 31822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:54:00.718254 31837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.719336 31822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:00.719435 31822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "3ae2a2095d42461cbad197de15847cd1"
format_stamp: "Formatted at 2026-09-26 14:54:00 on dist-test-slave-24mn"
server_key: "df69be153cdb6b246c9c6b8d7b19c10d"
server_key_iv: "d2e1525f75f0b957c29e18dc1caafb97"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.719528 31822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:00.728869 31822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:00.731657 31822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:00.731810 31822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:00.732337 31822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:00.733083 31822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:00.733127 31822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.733176 31822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:00.733201 31822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.740981 31822 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:34715
I20260926 14:54:00.741032 31950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:34715 every 8 connection(s)
I20260926 14:54:00.741770 31822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:54:00.743852 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 31822
I20260926 14:54:00.743976 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:54:00.744282 12563 external_mini_cluster.cc:1428] Setting key f543943f16f1410e46b641a75133eb27
Sep 26 14:54:00 dist-test-slave-24mn krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.193@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.194@KRBTEST.COM" created.
I20260926 14:54:00.756055 31799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.743362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:37913 (local address 127.12.68.254:42101)
0926 14:54:00.743481 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.743485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.744499 (+ 1014us) server_negotiation.cc:407] Connection header received
0926 14:54:00.746040 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.746043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.746084 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.746150 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:00.747528 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.748102 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.749614 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.749891 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.749987 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.752625 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:00.752628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:00.752630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:00.752652 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:00.754034 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.754537 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.754540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.754541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.754625 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.754922 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.754925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.754927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.755159 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:00.755243 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.755741 (+ 498us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.755867 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:54:00.756810 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.743788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42101 (local address 127.12.68.193:37913)
0926 14:54:00.744340 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.744399 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:00.745809 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:00.746342 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:00.746351 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:00.746771 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:00.747391 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.747409 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.748216 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.748220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.749470 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.749482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.750051 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.750055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.750257 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.750818 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:00.750838 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:00.752472 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:00.754187 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:00.754193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:00.754206 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:00.754443 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:00.754704 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:00.754707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:00.754709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:00.754839 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:00.755259 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:00.755265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:00.755506 (+ 241us) client_negotiation.cc:769] Sending connection context
0926 14:54:00.755684 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 14:54:00.756025 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":147,"threads_started":1}
I20260926 14:54:00.757987 31951 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42101
I20260926 14:54:00.758224 31951 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:00.758741 31951 heartbeater.cc:511] Master 127.12.68.254:42101 requested a full tablet report, sending...
I20260926 14:54:00.760349 31760 ts_manager.cc:194] Registered new tserver with Master: 3ae2a2095d42461cbad197de15847cd1 (127.12.68.193:34715)
I20260926 14:54:00.761440 31760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.193@KRBTEST.COM'} at 127.12.68.193:37913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
Entry for principal kudu/127.12.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab.
I20260926 14:54:00.768513 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42101
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:00.873412 31956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:00.873638 31956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:00.873685 31956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:00.877595 31956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:00.877645 31956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:00.877712 31956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:54:00.882761 31956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.194.keytab
--principal=kudu/127.12.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:00.883601 31956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:00.884492 31956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:00.893227 31962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:00.893219 31964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:00.893219 31961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:00.893424 31956 server_base.cc:1034] running on GCE node
I20260926 14:54:00.893891 31956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:00.894457 31956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:00.895607 31956 hybrid_clock.cc:648] HybridClock initialized: now 1790434440895590 us; error 31 us; skew 500 ppm
Sep 26 14:54:00 dist-test-slave-24mn krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:00.898094 31956 init.cc:377] Logged in from keytab as kudu/127.12.68.194@KRBTEST.COM (short username kudu)
I20260926 14:54:00.898888 31956 webserver.cc:460] Webserver started at http://127.12.68.194:37473/ using document root <none> and password file <none>
I20260926 14:54:00.899376 31956 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:00.899437 31956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:00.899607 31956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:00.901504 31956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "030109e66dcd496586798ef200124939"
format_stamp: "Formatted at 2026-09-26 14:54:00 on dist-test-slave-24mn"
server_key: "2bc207e3abfaacb8ba4b50f8aa2db02a"
server_key_iv: "a0e085ac626366a0c09e0ec599116b95"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.901996 31956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "030109e66dcd496586798ef200124939"
format_stamp: "Formatted at 2026-09-26 14:54:00 on dist-test-slave-24mn"
server_key: "2bc207e3abfaacb8ba4b50f8aa2db02a"
server_key_iv: "a0e085ac626366a0c09e0ec599116b95"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.905455 31956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:54:00.907675 31971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.908826 31956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:54:00.908929 31956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "030109e66dcd496586798ef200124939"
format_stamp: "Formatted at 2026-09-26 14:54:00 on dist-test-slave-24mn"
server_key: "2bc207e3abfaacb8ba4b50f8aa2db02a"
server_key_iv: "a0e085ac626366a0c09e0ec599116b95"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.909014 31956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:00.922945 31956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:00.925777 31956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:00.925927 31956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:00.926416 31956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:00.927134 31956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:00.927178 31956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.927225 31956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:00.927251 31956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.935228 31956 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:37499
I20260926 14:54:00.935264 32084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:37499 every 8 connection(s)
I20260926 14:54:00.936054 31956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:54:00.943598 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 31956
I20260926 14:54:00.943708 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:54:00.944015 12563 external_mini_cluster.cc:1428] Setting key 01e82dc981d0869290617ad280079a00
Sep 26 14:54:00 dist-test-slave-24mn krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.194@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:54:00.948601 31799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.937651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:60505 (local address 127.12.68.254:42101)
0926 14:54:00.937779 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.937782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.938525 (+ 743us) server_negotiation.cc:407] Connection header received
0926 14:54:00.939795 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.939798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.939851 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.939941 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:00.941012 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.941528 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.942616 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.942836 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.942921 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.945332 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:00.945336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:00.945339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:00.945362 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:00.946739 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.947220 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.947223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.947224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.947299 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.947591 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.947595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.947596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.947770 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:00.947852 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.948327 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.948443 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:54:00.949339 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.937989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42101 (local address 127.12.68.194:60505)
0926 14:54:00.938386 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.938423 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:00.939605 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:00.940126 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:00.940132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:00.940439 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:00.940873 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.940885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.941638 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.941643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.942489 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.942498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.942965 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.942969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.943130 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.943631 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:00.943647 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:00.945186 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:00.946851 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:00.946856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:00.946866 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:00.947101 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:00.947380 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:00.947383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:00.947385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:00.947508 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:00.947864 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:00.947870 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:00.948112 (+ 242us) client_negotiation.cc:769] Sending connection context
0926 14:54:00.948271 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 14:54:00.948568 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":108,"threads_started":1}
I20260926 14:54:00.950376 32085 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42101
I20260926 14:54:00.950629 32085 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:00.951191 32085 heartbeater.cc:511] Master 127.12.68.254:42101 requested a full tablet report, sending...
I20260926 14:54:00.952288 31760 ts_manager.cc:194] Registered new tserver with Master: 030109e66dcd496586798ef200124939 (127.12.68.194:37499)
WARNING: no policy specified for kudu/127.12.68.195@KRBTEST.COM; defaulting to no policy
I20260926 14:54:00.953050 31760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.194@KRBTEST.COM'} at 127.12.68.194:60505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
Entry for principal kudu/127.12.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab.
I20260926 14:54:00.968709 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42101
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:01.072921 32090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:01.073148 32090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:01.073187 32090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:01.077006 32090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:01.077054 32090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:01.077106 32090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:54:01.082180 32090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.195.keytab
--principal=kudu/127.12.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:42101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:01.083035 32090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:01.083962 32090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:01.092545 32096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:01.092535 32095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:01.092622 32090 server_base.cc:1034] running on GCE node
W20260926 14:54:01.092535 32098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:01.093120 32090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:01.093693 32090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:01.094842 32090 hybrid_clock.cc:648] HybridClock initialized: now 1790434441094826 us; error 29 us; skew 500 ppm
Sep 26 14:54:01 dist-test-slave-24mn krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434441, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:01.097237 32090 init.cc:377] Logged in from keytab as kudu/127.12.68.195@KRBTEST.COM (short username kudu)
I20260926 14:54:01.097982 32090 webserver.cc:460] Webserver started at http://127.12.68.195:45981/ using document root <none> and password file <none>
I20260926 14:54:01.098459 32090 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:01.098518 32090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:01.098693 32090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:01.100574 32090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "b50c172ab1e84603a7cd73d0bb57ba1a"
format_stamp: "Formatted at 2026-09-26 14:54:01 on dist-test-slave-24mn"
server_key: "b379faf54f2afc9c3bc765c30eef92ca"
server_key_iv: "442cc99a10366de5bebbbf66768788a2"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.101069 32090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "b50c172ab1e84603a7cd73d0bb57ba1a"
format_stamp: "Formatted at 2026-09-26 14:54:01 on dist-test-slave-24mn"
server_key: "b379faf54f2afc9c3bc765c30eef92ca"
server_key_iv: "442cc99a10366de5bebbbf66768788a2"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.104338 32090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:54:01.106460 32105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.107537 32090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:01.107633 32090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "b50c172ab1e84603a7cd73d0bb57ba1a"
format_stamp: "Formatted at 2026-09-26 14:54:01 on dist-test-slave-24mn"
server_key: "b379faf54f2afc9c3bc765c30eef92ca"
server_key_iv: "442cc99a10366de5bebbbf66768788a2"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.107723 32090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:01.136458 32090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:01.139132 32090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:01.139292 32090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:01.139827 32090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:01.140596 32090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:01.140642 32090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.140702 32090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:01.140728 32090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.148559 32090 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:41809
I20260926 14:54:01.148597 32218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:41809 every 8 connection(s)
I20260926 14:54:01.149353 32090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:54:01.153745 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 32090
I20260926 14:54:01.153853 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:54:01.154105 12563 external_mini_cluster.cc:1428] Setting key 9953d0df6500d6b611ed4fe924c5b8e0
Sep 26 14:54:01 dist-test-slave-24mn krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434441, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.195@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:54:01.161824 31799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.150927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:57045 (local address 127.12.68.254:42101)
0926 14:54:01.151043 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:01.151046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:01.151811 (+ 765us) server_negotiation.cc:407] Connection header received
0926 14:54:01.153062 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:01.153065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:01.153118 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:01.153219 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:01.154394 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.155062 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.156203 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.156426 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.156510 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.158625 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:01.158628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:01.158631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:01.158658 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:01.160023 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:01.160471 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:01.160474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:01.160476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:01.160553 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:01.160846 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:01.160849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:01.160851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:01.161023 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:01.161114 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:01.161551 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 14:54:01.161668 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:54:01.162532 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.151262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:42101 (local address 127.12.68.195:57045)
0926 14:54:01.151667 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.151706 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:01.152853 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:01.153391 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:01.153398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:01.153740 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:01.154249 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.154263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.155174 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.155177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.156094 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.156103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.156549 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.156552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.156713 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.157156 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:01.157174 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:01.158480 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:01.160123 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:01.160129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:01.160143 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:01.160378 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:01.160634 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:01.160637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:01.160639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:01.160762 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:01.161112 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:01.161118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:01.161340 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 14:54:01.161494 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 14:54:01.161798 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":109,"threads_started":1}
I20260926 14:54:01.163518 32219 heartbeater.cc:347] Connected to a master server at 127.12.68.254:42101
I20260926 14:54:01.163720 32219 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:01.164206 32219 heartbeater.cc:511] Master 127.12.68.254:42101 requested a full tablet report, sending...
I20260926 14:54:01.165107 31760 ts_manager.cc:194] Registered new tserver with Master: b50c172ab1e84603a7cd73d0bb57ba1a (127.12.68.195:41809)
I20260926 14:54:01.165635 31760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.68.195@KRBTEST.COM'} at 127.12.68.195:57045
I20260926 14:54:01.167572 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:54:01.175546 31799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.169380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.12.68.254:42101)
0926 14:54:01.169487 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:01.169490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:01.169627 (+ 137us) server_negotiation.cc:407] Connection header received
0926 14:54:01.169730 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:01.169732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:01.169765 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:01.169828 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:01.170420 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.170913 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.171676 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.171929 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.172032 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.173107 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:01.173110 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:01.173112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:01.173133 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:01.174366 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:01.174661 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:01.174663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:01.174664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:01.174728 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:01.174890 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:01.174892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:01.174893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:01.175101 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:01.175197 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:01.175347 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 14:54:01.175426 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:54:01.178153 31760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:54:01.181020 31760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:54:01.194052 32232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.188537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.12.68.195:41809)
0926 14:54:01.189836 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:01.189846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:01.189865 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:54:01.189930 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:01.189934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:01.190104 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:01.190468 (+ 364us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:01.190575 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.191274 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.192720 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.193446 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.193537 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.193674 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:01.193832 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 14:54:01.193922 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1206,"thread_start_us":120,"threads_started":1}
I20260926 14:54:01.194044 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.188398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:41809 (local address 127.0.0.1:42664)
0926 14:54:01.188953 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.188999 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:01.189137 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:01.190239 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:01.190242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:01.190256 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:01.190335 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.190339 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.191397 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.191399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.192616 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.192627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.193549 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.193553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.193664 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.193700 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 14:54:01.193797 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 14:54:01.193911 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"spinlock_wait_cycles":4736,"thread_start_us":71,"threads_started":1}
I20260926 14:54:01.194257 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.189110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:34715 (local address 127.0.0.1:51736)
0926 14:54:01.189360 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.189373 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:01.189456 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:01.190181 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:01.190184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:01.190203 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:01.190341 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.190347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.191785 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.191789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.192894 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.192901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.193887 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.193891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.193979 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.194005 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:54:01.194062 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:54:01.194129 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":68,"thread_start_us":73,"threads_started":1}
I20260926 14:54:01.194589 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.188519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:37499 (local address 127.0.0.1:52306)
0926 14:54:01.189021 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.189032 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:01.189137 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:01.190148 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:01.190155 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:01.190176 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:01.190331 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.190338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.191958 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.191961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.193088 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.193097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.194233 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.194236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.194321 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.194341 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:54:01.194402 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:54:01.194460 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":3,"spinlock_wait_cycles":6016,"thread_start_us":94,"threads_started":1}
I20260926 14:54:01.194808 32235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.189440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.12.68.193:34715)
0926 14:54:01.189772 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:01.189784 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:01.189806 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:54:01.189879 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:01.189884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:01.190048 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:01.190184 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:01.190459 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.191651 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.192996 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.193751 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.194415 (+ 664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.194544 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:01.194591 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 14:54:01.194681 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":136,"threads_started":1}
I20260926 14:54:01.194945 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.188908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.12.68.194:37499)
0926 14:54:01.189513 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:01.189525 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:01.189548 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:54:01.189626 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:01.189631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:01.189825 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:01.189972 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:01.190440 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.191651 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.193204 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.193964 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.194052 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.194176 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:01.194737 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 14:54:01.194827 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":126,"threads_started":1}
I20260926 14:54:01.197046 31885 tablet_service.cc:1467] Processing CreateTablet for tablet 84606cf1605e4231bea583f96ac41464 (DEFAULT_TABLE table=test-table [id=069591305f12404eb528fd8c0e72d0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:01.197067 32153 tablet_service.cc:1467] Processing CreateTablet for tablet 84606cf1605e4231bea583f96ac41464 (DEFAULT_TABLE table=test-table [id=069591305f12404eb528fd8c0e72d0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:01.197256 32019 tablet_service.cc:1467] Processing CreateTablet for tablet 84606cf1605e4231bea583f96ac41464 (DEFAULT_TABLE table=test-table [id=069591305f12404eb528fd8c0e72d0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:01.197959 31885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84606cf1605e4231bea583f96ac41464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:01.198031 32019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84606cf1605e4231bea583f96ac41464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:01.198041 32153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84606cf1605e4231bea583f96ac41464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:01.204345 32236 tablet_bootstrap.cc:492] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1: Bootstrap starting.
I20260926 14:54:01.205108 32237 tablet_bootstrap.cc:492] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a: Bootstrap starting.
I20260926 14:54:01.205108 32238 tablet_bootstrap.cc:492] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939: Bootstrap starting.
I20260926 14:54:01.206462 32236 tablet_bootstrap.cc:654] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:01.207127 32238 tablet_bootstrap.cc:654] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:01.207127 32237 tablet_bootstrap.cc:654] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:01.207479 32236 log.cc:824] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:01.208371 32238 log.cc:824] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:01.208371 32237 log.cc:824] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:01.209405 32236 tablet_bootstrap.cc:492] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1: No bootstrap required, opened a new log
I20260926 14:54:01.209594 32236 ts_tablet_manager.cc:1397] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:54:01.210064 32237 tablet_bootstrap.cc:492] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a: No bootstrap required, opened a new log
I20260926 14:54:01.210223 32237 ts_tablet_manager.cc:1397] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:54:01.210284 32238 tablet_bootstrap.cc:492] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939: No bootstrap required, opened a new log
I20260926 14:54:01.210449 32238 ts_tablet_manager.cc:1397] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:54:01.213241 32237 raft_consensus.cc:348] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41809 } } peers { permanent_uuid: "030109e66dcd496586798ef200124939" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 37499 } } peers { permanent_uuid: "3ae2a2095d42461cbad197de15847cd1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34715 } }
I20260926 14:54:01.213320 32236 raft_consensus.cc:348] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41809 } } peers { permanent_uuid: "030109e66dcd496586798ef200124939" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 37499 } } peers { permanent_uuid: "3ae2a2095d42461cbad197de15847cd1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34715 } }
I20260926 14:54:01.213487 32237 raft_consensus.cc:374] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:01.213542 32237 raft_consensus.cc:729] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50c172ab1e84603a7cd73d0bb57ba1a, State: Initialized, Role: FOLLOWER
I20260926 14:54:01.213554 32236 raft_consensus.cc:374] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:01.213599 32236 raft_consensus.cc:729] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae2a2095d42461cbad197de15847cd1, State: Initialized, Role: FOLLOWER
I20260926 14:54:01.213943 32237 consensus_queue.cc:260] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41809 } } peers { permanent_uuid: "030109e66dcd496586798ef200124939" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 37499 } } peers { permanent_uuid: "3ae2a2095d42461cbad197de15847cd1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34715 } }
I20260926 14:54:01.213986 32236 consensus_queue.cc:260] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41809 } } peers { permanent_uuid: "030109e66dcd496586798ef200124939" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 37499 } } peers { permanent_uuid: "3ae2a2095d42461cbad197de15847cd1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34715 } }
I20260926 14:54:01.214157 32238 raft_consensus.cc:348] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41809 } } peers { permanent_uuid: "030109e66dcd496586798ef200124939" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 37499 } } peers { permanent_uuid: "3ae2a2095d42461cbad197de15847cd1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34715 } }
I20260926 14:54:01.214388 32238 raft_consensus.cc:374] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:01.214442 32238 raft_consensus.cc:729] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 030109e66dcd496586798ef200124939, State: Initialized, Role: FOLLOWER
I20260926 14:54:01.214628 32219 heartbeater.cc:503] Master 127.12.68.254:42101 was elected leader, sending a full tablet report...
I20260926 14:54:01.214859 32237 ts_tablet_manager.cc:1428] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:54:01.214797 32238 consensus_queue.cc:260] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41809 } } peers { permanent_uuid: "030109e66dcd496586798ef200124939" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 37499 } } peers { permanent_uuid: "3ae2a2095d42461cbad197de15847cd1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34715 } }
I20260926 14:54:01.214872 32236 ts_tablet_manager.cc:1428] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:54:01.215178 31951 heartbeater.cc:503] Master 127.12.68.254:42101 was elected leader, sending a full tablet report...
I20260926 14:54:01.215551 32085 heartbeater.cc:503] Master 127.12.68.254:42101 was elected leader, sending a full tablet report...
I20260926 14:54:01.215631 32238 ts_tablet_manager.cc:1428] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:54:01.220158 32242 raft_consensus.cc:482] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:54:01.220290 32242 raft_consensus.cc:504] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41809 } } peers { permanent_uuid: "030109e66dcd496586798ef200124939" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 37499 } } peers { permanent_uuid: "3ae2a2095d42461cbad197de15847cd1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34715 } }
I20260926 14:54:01.221247 32242 leader_election.cc:288] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 030109e66dcd496586798ef200124939 (127.12.68.194:37499), 3ae2a2095d42461cbad197de15847cd1 (127.12.68.193:34715)
I20260926 14:54:01.225235 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.221412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:37499 (local address 127.12.68.195:34545)
0926 14:54:01.221527 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.221545 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:01.221632 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:01.221892 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:01.221895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:01.221910 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:01.221997 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.222001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.222880 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.222884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.223817 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.223826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.224668 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.224704 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.224837 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.224887 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 14:54:01.225018 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 14:54:01.225111 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 14:54:01.225557 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.221640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:34545 (local address 127.12.68.194:37499)
0926 14:54:01.221730 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:01.221733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:01.221744 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:54:01.221779 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:01.221781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:01.221815 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:01.221875 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:01.222098 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.222774 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.223944 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.224571 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.224681 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.224730 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:01.225381 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 14:54:01.225452 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:54:01.225695 32235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.221547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:45947 (local address 127.12.68.193:34715)
0926 14:54:01.221654 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:01.221658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:01.222155 (+ 497us) server_negotiation.cc:407] Connection header received
0926 14:54:01.222243 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:01.222245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:01.222280 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:01.222331 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:01.222568 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.223202 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.224377 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.224941 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.225037 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.225080 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:01.225534 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 14:54:01.225600 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:54:01.225986 32039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84606cf1605e4231bea583f96ac41464" candidate_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "030109e66dcd496586798ef200124939" is_pre_election: true
I20260926 14:54:01.226231 32245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.221413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:34715 (local address 127.12.68.195:45947)
0926 14:54:01.222068 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.222081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:01.222175 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:01.222353 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:01.222356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:01.222371 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:01.222489 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.222495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.223296 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.223298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.224279 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.224289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.225130 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.225133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.225230 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.225250 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:54:01.226024 (+ 774us) client_negotiation.cc:240] Negotiation successful
0926 14:54:01.226086 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":71,"thread_start_us":94,"threads_started":1}
I20260926 14:54:01.226221 32039 raft_consensus.cc:2454] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b50c172ab1e84603a7cd73d0bb57ba1a in term 0.
I20260926 14:54:01.226619 32107 leader_election.cc:302] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 030109e66dcd496586798ef200124939, b50c172ab1e84603a7cd73d0bb57ba1a; no voters:
I20260926 14:54:01.226740 31905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84606cf1605e4231bea583f96ac41464" candidate_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ae2a2095d42461cbad197de15847cd1" is_pre_election: true
I20260926 14:54:01.226869 32242 raft_consensus.cc:2781] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:54:01.226953 32242 raft_consensus.cc:482] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:54:01.226974 31905 raft_consensus.cc:2454] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b50c172ab1e84603a7cd73d0bb57ba1a in term 0.
I20260926 14:54:01.227000 32242 raft_consensus.cc:3037] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:01.228016 32242 raft_consensus.cc:504] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41809 } } peers { permanent_uuid: "030109e66dcd496586798ef200124939" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 37499 } } peers { permanent_uuid: "3ae2a2095d42461cbad197de15847cd1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34715 } }
I20260926 14:54:01.228308 32242 leader_election.cc:288] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [CANDIDATE]: Term 1 election: Requested vote from peers 030109e66dcd496586798ef200124939 (127.12.68.194:37499), 3ae2a2095d42461cbad197de15847cd1 (127.12.68.193:34715)
I20260926 14:54:01.228633 31905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84606cf1605e4231bea583f96ac41464" candidate_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ae2a2095d42461cbad197de15847cd1"
I20260926 14:54:01.228657 32039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84606cf1605e4231bea583f96ac41464" candidate_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "030109e66dcd496586798ef200124939"
I20260926 14:54:01.228744 31905 raft_consensus.cc:3037] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:01.228755 32039 raft_consensus.cc:3037] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:01.229658 32039 raft_consensus.cc:2454] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b50c172ab1e84603a7cd73d0bb57ba1a in term 1.
I20260926 14:54:01.229718 31905 raft_consensus.cc:2454] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b50c172ab1e84603a7cd73d0bb57ba1a in term 1.
I20260926 14:54:01.229911 32107 leader_election.cc:302] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 030109e66dcd496586798ef200124939, b50c172ab1e84603a7cd73d0bb57ba1a; no voters:
I20260926 14:54:01.230062 32242 raft_consensus.cc:2781] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:01.230259 32242 raft_consensus.cc:686] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [term 1 LEADER]: Becoming Leader. State: Replica: b50c172ab1e84603a7cd73d0bb57ba1a, State: Running, Role: LEADER
I20260926 14:54:01.230552 32242 consensus_queue.cc:237] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41809 } } peers { permanent_uuid: "030109e66dcd496586798ef200124939" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 37499 } } peers { permanent_uuid: "3ae2a2095d42461cbad197de15847cd1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34715 } }
I20260926 14:54:01.233477 31758 catalog_manager.cc:5495] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a reported cstate change: term changed from 0 to 1, leader changed from <none> to b50c172ab1e84603a7cd73d0bb57ba1a (127.12.68.195). New cstate: current_term: 1 leader_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50c172ab1e84603a7cd73d0bb57ba1a" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "030109e66dcd496586798ef200124939" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 37499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ae2a2095d42461cbad197de15847cd1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34715 } health_report { overall_health: UNKNOWN } } }
W20260926 14:54:01.242972 31952 tablet.cc:2369] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:54:01.243019 32232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.239918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.12.68.195:41809)
0926 14:54:01.240025 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:01.240028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:01.240040 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:54:01.240161 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:01.240163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:01.240208 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:01.240286 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:01.240446 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.240921 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.241756 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.241971 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.242048 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.242330 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:01.242693 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:01.242776 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:01.242832 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 14:54:01.242886 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:54:01.248057 32039 raft_consensus.cc:1261] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939 [term 1 FOLLOWER]: Refusing update from remote peer b50c172ab1e84603a7cd73d0bb57ba1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:01.248057 31905 raft_consensus.cc:1261] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1 [term 1 FOLLOWER]: Refusing update from remote peer b50c172ab1e84603a7cd73d0bb57ba1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:01.248651 32246 consensus_queue.cc:1035] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ae2a2095d42461cbad197de15847cd1" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 34715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:01.248802 32242 consensus_queue.cc:1035] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "030109e66dcd496586798ef200124939" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 37499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:01.259307 32251 mvcc.cc:205] Tried to move back new op lower bound from 7333619471346016256 to 7333619471282778112. Current Snapshot: MvccSnapshot[applied={T|T < 7333619471346016256 or (T in {7333619471346016256})}]
I20260926 14:54:01.262571 31757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:54:01.262701 31757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:54:01.264654 31757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37dd2e69157848fe9da0f6b304f8f863: Sending DeleteTablet for 3 replicas of tablet 84606cf1605e4231bea583f96ac41464
I20260926 14:54:01.265305 31885 tablet_service.cc:1514] Processing DeleteTablet for tablet 84606cf1605e4231bea583f96ac41464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:01 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:51736
I20260926 14:54:01.265321 32019 tablet_service.cc:1514] Processing DeleteTablet for tablet 84606cf1605e4231bea583f96ac41464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:01 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:52306
I20260926 14:54:01.265681 32259 tablet_replica.cc:330] T 84606cf1605e4231bea583f96ac41464 P 3ae2a2095d42461cbad197de15847cd1: stopping tablet replica
I20260926 14:54:01.265821 32153 tablet_service.cc:1514] Processing DeleteTablet for tablet 84606cf1605e4231bea583f96ac41464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:01 UTC) from {username='kudu', principal='kudu/127.12.68.254@KRBTEST.COM'} at 127.0.0.1:42664
I20260926 14:54:01.265937 32260 tablet_replica.cc:330] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939: stopping tablet replica
I20260926 14:54:01.266179 32261 tablet_replica.cc:330] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a: stopping tablet replica
I20260926 14:54:01.266204 32260 raft_consensus.cc:2229] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:54:01.266407 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 31822
I20260926 14:54:01.266426 32261 raft_consensus.cc:2229] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:54:01.266471 32260 raft_consensus.cc:2258] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:54:01.268172 32261 raft_consensus.cc:2258] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:54:01.268692 32260 ts_tablet_manager.cc:1905] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:01.269423 32261 ts_tablet_manager.cc:1905] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:01.271286 32260 ts_tablet_manager.cc:1918] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:54:01.271379 32260 log.cc:1196] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/84606cf1605e4231bea583f96ac41464
I20260926 14:54:01.271690 32260 ts_tablet_manager.cc:1939] T 84606cf1605e4231bea583f96ac41464 P 030109e66dcd496586798ef200124939: Deleting consensus metadata
I20260926 14:54:01.271816 32261 ts_tablet_manager.cc:1918] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:54:01.271919 32261 log.cc:1196] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/84606cf1605e4231bea583f96ac41464
I20260926 14:54:01.272210 32261 ts_tablet_manager.cc:1939] T 84606cf1605e4231bea583f96ac41464 P b50c172ab1e84603a7cd73d0bb57ba1a: Deleting consensus metadata
I20260926 14:54:01.272818 31745 catalog_manager.cc:4841] TS 030109e66dcd496586798ef200124939 (127.12.68.194:37499): tablet 84606cf1605e4231bea583f96ac41464 (table test-table [id=069591305f12404eb528fd8c0e72d0f1]) successfully deleted
I20260926 14:54:01.273120 31744 catalog_manager.cc:4841] TS b50c172ab1e84603a7cd73d0bb57ba1a (127.12.68.195:41809): tablet 84606cf1605e4231bea583f96ac41464 (table test-table [id=069591305f12404eb528fd8c0e72d0f1]) successfully deleted
W20260926 14:54:01.274166 31744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:34715 (error 108)
I20260926 14:54:01.274364 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 31956
I20260926 14:54:01.275601 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.275346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:34715 (local address 127.0.0.1:51740)
0926 14:54:01.275462 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.275523 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:34715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 14:54:01.275813 31744 catalog_manager.cc:4584] TS 3ae2a2095d42461cbad197de15847cd1 (127.12.68.193:34715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84606cf1605e4231bea583f96ac41464: Network error: Client connection negotiation failed: client connection to 127.12.68.193:34715: connect: Connection refused (error 111)
I20260926 14:54:01.279601 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 32090
I20260926 14:54:01.284312 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 31728
2026-09-26T14:54:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1998 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T14:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:01Z Disabled control of system clock
I20260926 14:54:01.310662 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:45973
--webserver_interface=127.12.68.254
--webserver_port=0
--builtin_ntp_servers=127.12.68.212:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:45973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:54:01.412734 32266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:01.412959 32266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:01.412992 32266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:01.416867 32266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:01.416913 32266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:01.416927 32266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:01.416942 32266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:01.416954 32266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:01.421470 32266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:45973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:45973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:01.422343 32266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:01.423539 32266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:01.431250 32271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:01.431274 32272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:01.431286 32274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:01.431419 32266 server_base.cc:1034] running on GCE node
I20260926 14:54:01.432287 32266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:01.433254 32266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:01.434399 32266 hybrid_clock.cc:648] HybridClock initialized: now 1790434441434378 us; error 35 us; skew 500 ppm
I20260926 14:54:01.436100 32266 webserver.cc:460] Webserver started at http://127.12.68.254:43387/ using document root <none> and password file <none>
I20260926 14:54:01.436620 32266 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:01.436676 32266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:01.436858 32266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:01.438786 32266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "8805a85b67704f62b8b24c98cca9ded2"
format_stamp: "Formatted at 2026-09-26 14:54:01 on dist-test-slave-24mn"
server_key: "f36250dee0e7b227f6bd2e03e2490b34"
server_key_iv: "9d3bc39d517fe37be5bea43dfaf817df"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.439287 32266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "8805a85b67704f62b8b24c98cca9ded2"
format_stamp: "Formatted at 2026-09-26 14:54:01 on dist-test-slave-24mn"
server_key: "f36250dee0e7b227f6bd2e03e2490b34"
server_key_iv: "9d3bc39d517fe37be5bea43dfaf817df"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.442693 32266 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:54:01.444933 32280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.446048 32266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:54:01.446148 32266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "8805a85b67704f62b8b24c98cca9ded2"
format_stamp: "Formatted at 2026-09-26 14:54:01 on dist-test-slave-24mn"
server_key: "f36250dee0e7b227f6bd2e03e2490b34"
server_key_iv: "9d3bc39d517fe37be5bea43dfaf817df"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.446228 32266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:01.472078 32266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:01.472733 32266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:01.472879 32266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:01.479327 32266 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:45973
I20260926 14:54:01.479336 32332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:45973 every 8 connection(s)
I20260926 14:54:01.480239 32266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:01.483131 32333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:01.485404 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 32266
I20260926 14:54:01.485507 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:01.485770 12563 external_mini_cluster.cc:1428] Setting key d9487af4cacd980ddc970429c863211e
I20260926 14:54:01.488829 32333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2: Bootstrap starting.
I20260926 14:54:01.490890 32333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:01.491572 32333 log.cc:824] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:01.493381 32333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2: No bootstrap required, opened a new log
I20260926 14:54:01.494719 32336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.487112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.12.68.254:45973)
0926 14:54:01.487717 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:01.487732 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:01.487766 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:54:01.488448 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:01.488471 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:01.488858 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:01.489174 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:01.489582 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.490526 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.491424 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.491941 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.492078 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.492877 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:01.492882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:01.492894 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:01.493264 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:01.493317 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:01.493336 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:01.493415 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:01.493573 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 14:54:01.493839 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
I20260926 14:54:01.497004 32333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8805a85b67704f62b8b24c98cca9ded2" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45973 } }
I20260926 14:54:01.497208 32333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:01.497246 32333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8805a85b67704f62b8b24c98cca9ded2, State: Initialized, Role: FOLLOWER
I20260926 14:54:01.497575 32333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8805a85b67704f62b8b24c98cca9ded2" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45973 } }
I20260926 14:54:01.497687 32333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:01.497727 32333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:01.497792 32333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:01.498535 32333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8805a85b67704f62b8b24c98cca9ded2" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45973 } }
I20260926 14:54:01.498817 32333 leader_election.cc:302] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8805a85b67704f62b8b24c98cca9ded2; no voters:
I20260926 14:54:01.499039 32333 leader_election.cc:288] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:01.499181 32338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:01.499390 32338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [term 1 LEADER]: Becoming Leader. State: Replica: 8805a85b67704f62b8b24c98cca9ded2, State: Running, Role: LEADER
I20260926 14:54:01.499673 32338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8805a85b67704f62b8b24c98cca9ded2" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45973 } }
I20260926 14:54:01.500195 32333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:01.501293 32340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8805a85b67704f62b8b24c98cca9ded2. Latest consensus state: current_term: 1 leader_uuid: "8805a85b67704f62b8b24c98cca9ded2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8805a85b67704f62b8b24c98cca9ded2" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45973 } } }
I20260926 14:54:01.501317 32339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8805a85b67704f62b8b24c98cca9ded2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8805a85b67704f62b8b24c98cca9ded2" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 45973 } } }
I20260926 14:54:01.501495 32340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:01.501523 32339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:01.502369 32350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:01.504802 32350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:01.509459 32350 catalog_manager.cc:1347] Generated new cluster ID: adea3e36f75a478bb5c2b3b7bd79f39a
I20260926 14:54:01.509517 32350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:01.519171 32350 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:01.520030 32350 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:01.525301 32350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2: Generated new TSK 0
I20260926 14:54:01.525872 32350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:01.530390 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.193:0
--local_ip_for_outbound_sockets=127.12.68.193
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45973
--builtin_ntp_servers=127.12.68.212:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:54:01.635182 32357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:01.635404 32357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:01.635439 32357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:01.639379 32357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:01.639426 32357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:01.639477 32357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.193
I20260926 14:54:01.644081 32357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.68.193
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.193
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:01.645018 32357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:01.646297 32357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:54:01.655378 32357 server_base.cc:1034] running on GCE node
W20260926 14:54:01.655339 32362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:01.655342 32363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:01.655339 32365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:01.655910 32357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:01.656466 32357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:01.657618 32357 hybrid_clock.cc:648] HybridClock initialized: now 1790434441657591 us; error 38 us; skew 500 ppm
I20260926 14:54:01.659205 32357 webserver.cc:460] Webserver started at http://127.12.68.193:35843/ using document root <none> and password file <none>
I20260926 14:54:01.659735 32357 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:01.659795 32357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:01.660008 32357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:01.661916 32357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/data/instance:
uuid: "697aabe47da4492b87a75d405eb04bdd"
format_stamp: "Formatted at 2026-09-26 14:54:01 on dist-test-slave-24mn"
server_key: "95fa125ea881c7fd10e55624dfe6682c"
server_key_iv: "edf877bcb3d073d11cf3c910b6c0e3b6"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.662416 32357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance:
uuid: "697aabe47da4492b87a75d405eb04bdd"
format_stamp: "Formatted at 2026-09-26 14:54:01 on dist-test-slave-24mn"
server_key: "95fa125ea881c7fd10e55624dfe6682c"
server_key_iv: "edf877bcb3d073d11cf3c910b6c0e3b6"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.665844 32357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:54:01.668044 32371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.669147 32357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:54:01.669253 32357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/wal
uuid: "697aabe47da4492b87a75d405eb04bdd"
format_stamp: "Formatted at 2026-09-26 14:54:01 on dist-test-slave-24mn"
server_key: "95fa125ea881c7fd10e55624dfe6682c"
server_key_iv: "edf877bcb3d073d11cf3c910b6c0e3b6"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.669337 32357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:01.681267 32357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:01.681843 32357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:01.681998 32357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:01.682507 32357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:01.683351 32357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:01.683396 32357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.683444 32357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:01.683473 32357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.691583 32357 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.193:39717
I20260926 14:54:01.691648 32484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.193:39717 every 8 connection(s)
I20260926 14:54:01.692423 32357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/data/info.pb
I20260926 14:54:01.696238 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 32357
I20260926 14:54:01.696372 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-0/wal/instance
I20260926 14:54:01.696689 12563 external_mini_cluster.cc:1428] Setting key bfd0387482abedd73acf7c0ef5cc4206
I20260926 14:54:01.698478 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.194:0
--local_ip_for_outbound_sockets=127.12.68.194
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45973
--builtin_ntp_servers=127.12.68.212:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:54:01.701798 32336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.694118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:57805 (local address 127.12.68.254:45973)
0926 14:54:01.694241 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:01.694244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:01.695236 (+ 992us) server_negotiation.cc:407] Connection header received
0926 14:54:01.696820 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:01.696824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:01.696870 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:01.696934 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:01.697937 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.698514 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.700039 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.700268 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.700377 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.701346 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:01.701349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:01.701351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:01.701362 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:01.701381 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:01.701389 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:01.701444 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:01.701604 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 14:54:01.701661 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:54:01.702605 32487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.694478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45973 (local address 127.12.68.193:57805)
0926 14:54:01.695107 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.695144 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:01.696568 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:01.697174 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:01.697180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:01.697284 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:01.697798 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.697815 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.698651 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.698654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.699924 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.699935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.700405 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.700410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.700579 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.701111 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:54:01.701113 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:01.701150 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:54:01.701210 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:01.701478 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:01.701486 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:01.701492 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:54:01.701590 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 14:54:01.701845 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260926 14:54:01.703727 32485 heartbeater.cc:347] Connected to a master server at 127.12.68.254:45973
I20260926 14:54:01.703951 32485 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:01.704427 32485 heartbeater.cc:511] Master 127.12.68.254:45973 requested a full tablet report, sending...
I20260926 14:54:01.705981 32297 ts_manager.cc:194] Registered new tserver with Master: 697aabe47da4492b87a75d405eb04bdd (127.12.68.193:39717)
I20260926 14:54:01.707549 32297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.68.193:57805
W20260926 14:54:01.803937 32488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:01.804177 32488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:01.804214 32488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:01.808072 32488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:01.808113 32488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:01.808162 32488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.194
I20260926 14:54:01.812686 32488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.68.194
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.194
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:01.813611 32488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:01.814837 32488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:01.823652 32493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:01.823748 32496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:01.823787 32488 server_base.cc:1034] running on GCE node
W20260926 14:54:01.823689 32494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:01.824172 32488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:01.824781 32488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:01.825932 32488 hybrid_clock.cc:648] HybridClock initialized: now 1790434441825898 us; error 43 us; skew 500 ppm
I20260926 14:54:01.827611 32488 webserver.cc:460] Webserver started at http://127.12.68.194:34433/ using document root <none> and password file <none>
I20260926 14:54:01.828222 32488 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:01.828303 32488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:01.828498 32488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:01.830428 32488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/data/instance:
uuid: "d842933ab12d49c58360ac129e5ebeaf"
format_stamp: "Formatted at 2026-09-26 14:54:01 on dist-test-slave-24mn"
server_key: "6ba7fc268db05afc1d56b69919ccf3a0"
server_key_iv: "d4a7f2a6ac9a3d8c1215f788262fe442"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.830947 32488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance:
uuid: "d842933ab12d49c58360ac129e5ebeaf"
format_stamp: "Formatted at 2026-09-26 14:54:01 on dist-test-slave-24mn"
server_key: "6ba7fc268db05afc1d56b69919ccf3a0"
server_key_iv: "d4a7f2a6ac9a3d8c1215f788262fe442"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.834491 32488 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 14:54:01.836776 32502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.837939 32488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:54:01.838028 32488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/wal
uuid: "d842933ab12d49c58360ac129e5ebeaf"
format_stamp: "Formatted at 2026-09-26 14:54:01 on dist-test-slave-24mn"
server_key: "6ba7fc268db05afc1d56b69919ccf3a0"
server_key_iv: "d4a7f2a6ac9a3d8c1215f788262fe442"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.838135 32488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:01.855150 32488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:01.855772 32488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:01.855945 32488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:01.856482 32488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:01.857378 32488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:01.857429 32488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.857479 32488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:01.857508 32488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.865787 32488 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.194:44835
I20260926 14:54:01.865824 32615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.194:44835 every 8 connection(s)
I20260926 14:54:01.866644 32488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/data/info.pb
I20260926 14:54:01.874634 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 32488
I20260926 14:54:01.874760 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/wal/instance
I20260926 14:54:01.875036 12563 external_mini_cluster.cc:1428] Setting key 418dd60ca79a70d6377c9cb333e6d98a
I20260926 14:54:01.875464 32336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.868348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.194:56253 (local address 127.12.68.254:45973)
0926 14:54:01.868461 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:01.868465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:01.869565 (+ 1100us) server_negotiation.cc:407] Connection header received
0926 14:54:01.870885 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:01.870888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:01.870937 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:01.871027 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:01.871923 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.872462 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.873631 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.873866 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.873956 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.875010 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:01.875013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:01.875016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:01.875030 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:01.875048 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:01.875058 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:01.875120 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:01.875264 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 14:54:01.875324 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:54:01.876417 32618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.868712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45973 (local address 127.12.68.194:56253)
0926 14:54:01.869412 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.869453 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:01.870671 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:01.871197 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:01.871205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:01.871287 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:01.871775 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.871787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.872589 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.872592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.873520 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.873531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.873983 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.873985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.874124 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.874739 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:54:01.874741 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:01.874793 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:54:01.874869 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:01.875130 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:01.875138 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:01.875143 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:54:01.875252 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 14:54:01.875523 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":105,"threads_started":1}
I20260926 14:54:01.876907 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.68.195:0
--local_ip_for_outbound_sockets=127.12.68.195
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45973
--builtin_ntp_servers=127.12.68.212:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:54:01.877435 32616 heartbeater.cc:347] Connected to a master server at 127.12.68.254:45973
I20260926 14:54:01.877645 32616 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:01.878101 32616 heartbeater.cc:511] Master 127.12.68.254:45973 requested a full tablet report, sending...
I20260926 14:54:01.879026 32297 ts_manager.cc:194] Registered new tserver with Master: d842933ab12d49c58360ac129e5ebeaf (127.12.68.194:44835)
I20260926 14:54:01.879575 32297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.68.194:56253
W20260926 14:54:01.982494 32619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:01.982735 32619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:01.982774 32619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:01.986675 32619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:01.986726 32619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:01.986780 32619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.68.195
I20260926 14:54:01.991386 32619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.68.195
--webserver_port=0
--tserver_master_addrs=127.12.68.254:45973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.68.195
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:01.992434 32619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:01.993716 32619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:02.002740 32627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:02.002735 32624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:02.002769 32625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:02.003051 32619 server_base.cc:1034] running on GCE node
I20260926 14:54:02.003482 32619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:02.004084 32619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:02.005260 32619 hybrid_clock.cc:648] HybridClock initialized: now 1790434442005242 us; error 29 us; skew 500 ppm
I20260926 14:54:02.006973 32619 webserver.cc:460] Webserver started at http://127.12.68.195:36635/ using document root <none> and password file <none>
I20260926 14:54:02.007496 32619 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:02.007570 32619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:02.007747 32619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:02.009711 32619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/data/instance:
uuid: "0106846b4b4e4e74a75ba712a907b4e9"
format_stamp: "Formatted at 2026-09-26 14:54:02 on dist-test-slave-24mn"
server_key: "8e5a2d8d25b50386c8b84b95bb39a879"
server_key_iv: "9216120a0460cb148d79777217cd997a"
server_key_version: "encryptionkey@0"
I20260926 14:54:02.010226 32619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance:
uuid: "0106846b4b4e4e74a75ba712a907b4e9"
format_stamp: "Formatted at 2026-09-26 14:54:02 on dist-test-slave-24mn"
server_key: "8e5a2d8d25b50386c8b84b95bb39a879"
server_key_iv: "9216120a0460cb148d79777217cd997a"
server_key_version: "encryptionkey@0"
I20260926 14:54:02.013697 32619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 14:54:02.015986 32633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:02.017167 32619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:02.017275 32619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/wal
uuid: "0106846b4b4e4e74a75ba712a907b4e9"
format_stamp: "Formatted at 2026-09-26 14:54:02 on dist-test-slave-24mn"
server_key: "8e5a2d8d25b50386c8b84b95bb39a879"
server_key_iv: "9216120a0460cb148d79777217cd997a"
server_key_version: "encryptionkey@0"
I20260926 14:54:02.017372 32619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:02.048931 32619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:02.049638 32619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:02.049799 32619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:02.050375 32619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:02.051327 32619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:02.051375 32619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:02.051427 32619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:02.051455 32619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:02.059616 32619 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.195:41637
I20260926 14:54:02.059656 32746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.195:41637 every 8 connection(s)
I20260926 14:54:02.060521 32619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/data/info.pb
I20260926 14:54:02.062749 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 32619
I20260926 14:54:02.062839 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/wal/instance
I20260926 14:54:02.063187 12563 external_mini_cluster.cc:1428] Setting key a47007a70f9f29ace29261bf91138253
I20260926 14:54:02.069492 32336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.062253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.195:43475 (local address 127.12.68.254:45973)
0926 14:54:02.062412 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:02.062416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:02.063594 (+ 1178us) server_negotiation.cc:407] Connection header received
0926 14:54:02.065175 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:02.065177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:02.065229 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:02.065298 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:02.066162 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.066671 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.067752 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.068008 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.068102 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.069055 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:02.069059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:02.069062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:02.069074 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:02.069094 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:02.069101 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:02.069157 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:02.069294 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 14:54:02.069354 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:54:02.070298 32749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.062664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:45973 (local address 127.12.68.195:43475)
0926 14:54:02.063425 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:02.063479 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:02.064986 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:02.065463 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:02.065470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:02.065564 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:02.066032 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.066044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.066779 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.066783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.067642 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.067650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.068146 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.068150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.068303 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.068813 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:54:02.068815 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:02.068853 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:54:02.068913 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:02.069166 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:02.069174 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:02.069180 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:54:02.069285 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 14:54:02.069575 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":133,"threads_started":1}
I20260926 14:54:02.071311 32747 heartbeater.cc:347] Connected to a master server at 127.12.68.254:45973
I20260926 14:54:02.071537 32747 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:02.072050 32747 heartbeater.cc:511] Master 127.12.68.254:45973 requested a full tablet report, sending...
I20260926 14:54:02.073058 32297 ts_manager.cc:194] Registered new tserver with Master: 0106846b4b4e4e74a75ba712a907b4e9 (127.12.68.195:41637)
I20260926 14:54:02.073657 32297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.68.195:43475
I20260926 14:54:02.076757 12563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:54:02.081826 32336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.078599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.12.68.254:45973)
0926 14:54:02.078701 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:02.078704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:02.078784 (+ 80us) server_negotiation.cc:407] Connection header received
0926 14:54:02.078879 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:02.078881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:02.078916 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:02.078981 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:02.079189 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.079676 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.080493 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.080699 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.080772 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.081499 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:02.081502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:02.081504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:02.081514 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:02.081544 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:02.081551 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:02.081606 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:02.081677 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 14:54:02.081723 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:54:02.084466 32297 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:54:02.087332 32297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:54:02.101142 32762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.096091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.12.68.195:41637)
0926 14:54:02.096350 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:02.096360 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:02.096385 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:54:02.096460 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:02.096466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:02.096673 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:02.097222 (+ 549us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:02.097355 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.098209 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.099834 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.100488 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.100584 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.100776 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:02.100882 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 14:54:02.100975 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
I20260926 14:54:02.101136 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.095405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:41637 (local address 127.0.0.1:46036)
0926 14:54:02.096045 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:02.096096 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:02.096218 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:02.096862 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:02.096868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:02.096905 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:02.097066 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.097073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.098306 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.098309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.099719 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.099725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.100597 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.100601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.100707 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.100742 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 14:54:02.100854 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 14:54:02.100976 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":46,"spinlock_wait_cycles":6400,"thread_start_us":76,"threads_started":1}
I20260926 14:54:02.101287 32761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.096068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:44835 (local address 127.0.0.1:37504)
0926 14:54:02.096431 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:02.096443 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:02.096517 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:02.097177 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:02.097180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:02.097197 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:02.097292 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.097299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.098517 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.098520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.099714 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.099725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.100875 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.100879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.100962 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.100983 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:54:02.101029 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 14:54:02.101095 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260926 14:54:02.101687 32760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.095484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53704 (local address 127.12.68.193:39717)
0926 14:54:02.095798 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:02.095814 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:02.096834 (+ 1020us) server_negotiation.cc:407] Connection header received
0926 14:54:02.097079 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:02.097085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:02.097281 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:02.097421 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:02.097793 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.098772 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.100008 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.100803 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.100926 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.101104 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:02.101431 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 14:54:02.101537 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":113,"threads_started":1}
I20260926 14:54:02.102046 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.095347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:39717 (local address 127.0.0.1:53704)
0926 14:54:02.096744 (+ 1397us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:02.096758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:02.096939 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:02.097557 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:02.097560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:02.097576 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:02.097667 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.097672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.098871 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.098875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.099874 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.099918 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.101172 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.101176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.101257 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.101277 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:54:02.101338 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:54:02.101901 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":1,"spinlock_wait_cycles":69504,"thread_start_us":56,"threads_started":1}
I20260926 14:54:02.102823 32763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.096458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.12.68.194:44835)
0926 14:54:02.096758 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:02.096768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:02.096792 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:54:02.096865 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:02.096870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:02.097060 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:02.097189 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:02.097392 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.098405 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.099963 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.100738 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.101626 (+ 888us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.101940 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:02.101988 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 14:54:02.102073 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":128,"threads_started":1}
I20260926 14:54:02.104501 32681 tablet_service.cc:1467] Processing CreateTablet for tablet 39c9126b4fbc48b48bc5c99cfb00ad9d (DEFAULT_TABLE table=test-table [id=452dc35060d74fee9119b2af9a7aea89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:02.104501 32419 tablet_service.cc:1467] Processing CreateTablet for tablet 39c9126b4fbc48b48bc5c99cfb00ad9d (DEFAULT_TABLE table=test-table [id=452dc35060d74fee9119b2af9a7aea89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:02.105077 32550 tablet_service.cc:1467] Processing CreateTablet for tablet 39c9126b4fbc48b48bc5c99cfb00ad9d (DEFAULT_TABLE table=test-table [id=452dc35060d74fee9119b2af9a7aea89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:02.105612 32419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c9126b4fbc48b48bc5c99cfb00ad9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:02.105612 32681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c9126b4fbc48b48bc5c99cfb00ad9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:02.105918 32550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c9126b4fbc48b48bc5c99cfb00ad9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:02.111748 32764 tablet_bootstrap.cc:492] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf: Bootstrap starting.
I20260926 14:54:02.113306 32766 tablet_bootstrap.cc:492] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9: Bootstrap starting.
I20260926 14:54:02.113912 32764 tablet_bootstrap.cc:654] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:02.114588 32764 log.cc:824] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:02.115072 32766 tablet_bootstrap.cc:654] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:02.115191 32765 tablet_bootstrap.cc:492] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd: Bootstrap starting.
I20260926 14:54:02.115909 32766 log.cc:824] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:02.116675 32764 tablet_bootstrap.cc:492] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf: No bootstrap required, opened a new log
I20260926 14:54:02.116873 32764 ts_tablet_manager.cc:1397] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:54:02.117236 32765 tablet_bootstrap.cc:654] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:02.117957 32766 tablet_bootstrap.cc:492] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9: No bootstrap required, opened a new log
I20260926 14:54:02.118136 32766 ts_tablet_manager.cc:1397] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:54:02.118121 32765 log.cc:824] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:02.120065 32765 tablet_bootstrap.cc:492] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd: No bootstrap required, opened a new log
I20260926 14:54:02.120249 32765 ts_tablet_manager.cc:1397] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:54:02.120362 32764 raft_consensus.cc:348] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697aabe47da4492b87a75d405eb04bdd" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 39717 } } peers { permanent_uuid: "0106846b4b4e4e74a75ba712a907b4e9" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41637 } } peers { permanent_uuid: "d842933ab12d49c58360ac129e5ebeaf" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 44835 } }
I20260926 14:54:02.120584 32764 raft_consensus.cc:374] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:02.120642 32764 raft_consensus.cc:729] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d842933ab12d49c58360ac129e5ebeaf, State: Initialized, Role: FOLLOWER
I20260926 14:54:02.121165 32764 consensus_queue.cc:260] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697aabe47da4492b87a75d405eb04bdd" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 39717 } } peers { permanent_uuid: "0106846b4b4e4e74a75ba712a907b4e9" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41637 } } peers { permanent_uuid: "d842933ab12d49c58360ac129e5ebeaf" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 44835 } }
I20260926 14:54:02.121832 32616 heartbeater.cc:503] Master 127.12.68.254:45973 was elected leader, sending a full tablet report...
I20260926 14:54:02.122125 32766 raft_consensus.cc:348] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697aabe47da4492b87a75d405eb04bdd" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 39717 } } peers { permanent_uuid: "0106846b4b4e4e74a75ba712a907b4e9" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41637 } } peers { permanent_uuid: "d842933ab12d49c58360ac129e5ebeaf" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 44835 } }
I20260926 14:54:02.122375 32766 raft_consensus.cc:374] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:02.122428 32766 raft_consensus.cc:729] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0106846b4b4e4e74a75ba712a907b4e9, State: Initialized, Role: FOLLOWER
I20260926 14:54:02.122849 32766 consensus_queue.cc:260] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697aabe47da4492b87a75d405eb04bdd" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 39717 } } peers { permanent_uuid: "0106846b4b4e4e74a75ba712a907b4e9" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41637 } } peers { permanent_uuid: "d842933ab12d49c58360ac129e5ebeaf" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 44835 } }
I20260926 14:54:02.123123 32764 ts_tablet_manager.cc:1428] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:54:02.123765 32766 ts_tablet_manager.cc:1428] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:54:02.124182 32747 heartbeater.cc:503] Master 127.12.68.254:45973 was elected leader, sending a full tablet report...
I20260926 14:54:02.124248 32765 raft_consensus.cc:348] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697aabe47da4492b87a75d405eb04bdd" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 39717 } } peers { permanent_uuid: "0106846b4b4e4e74a75ba712a907b4e9" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41637 } } peers { permanent_uuid: "d842933ab12d49c58360ac129e5ebeaf" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 44835 } }
I20260926 14:54:02.124413 32765 raft_consensus.cc:374] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:02.124445 32765 raft_consensus.cc:729] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697aabe47da4492b87a75d405eb04bdd, State: Initialized, Role: FOLLOWER
I20260926 14:54:02.124758 32765 consensus_queue.cc:260] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697aabe47da4492b87a75d405eb04bdd" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 39717 } } peers { permanent_uuid: "0106846b4b4e4e74a75ba712a907b4e9" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41637 } } peers { permanent_uuid: "d842933ab12d49c58360ac129e5ebeaf" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 44835 } }
I20260926 14:54:02.125651 32765 ts_tablet_manager.cc:1428] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:54:02.125950 32485 heartbeater.cc:503] Master 127.12.68.254:45973 was elected leader, sending a full tablet report...
I20260926 14:54:02.128942 304 raft_consensus.cc:482] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:54:02.129050 304 raft_consensus.cc:504] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697aabe47da4492b87a75d405eb04bdd" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 39717 } } peers { permanent_uuid: "0106846b4b4e4e74a75ba712a907b4e9" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41637 } } peers { permanent_uuid: "d842933ab12d49c58360ac129e5ebeaf" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 44835 } }
I20260926 14:54:02.129995 304 leader_election.cc:288] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0106846b4b4e4e74a75ba712a907b4e9 (127.12.68.195:41637), d842933ab12d49c58360ac129e5ebeaf (127.12.68.194:44835)
I20260926 14:54:02.134972 32487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.130475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.195:41637 (local address 127.12.68.193:59221)
0926 14:54:02.131071 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:02.131095 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:02.131382 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:02.131756 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:02.131759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:02.131777 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:02.131986 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.131996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.132702 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.132705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.133665 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.133674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.134486 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.134498 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.134665 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.134691 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:54:02.134764 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:54:02.134836 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":37,"spinlock_wait_cycles":39168}
I20260926 14:54:02.135311 32762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.130967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:59221 (local address 127.12.68.195:41637)
0926 14:54:02.131169 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:02.131174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:02.131311 (+ 137us) server_negotiation.cc:407] Connection header received
0926 14:54:02.131503 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:02.131506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:02.131556 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:02.131632 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:02.132119 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.132600 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.133771 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.134330 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.134443 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.134494 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:02.135128 (+ 634us) server_negotiation.cc:299] Negotiation successful
0926 14:54:02.135201 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 14:54:02.135346 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.130711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.194:44835 (local address 127.12.68.193:41123)
0926 14:54:02.131035 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:02.131051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:02.131157 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:02.131744 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:02.131747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:02.131763 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:02.131875 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.131906 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.132674 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.132677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.133910 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.133921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.134848 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.134851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.134947 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.134968 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:54:02.135035 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:54:02.135205 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":34,"thread_start_us":103,"threads_started":1}
I20260926 14:54:02.136106 32701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c9126b4fbc48b48bc5c99cfb00ad9d" candidate_uuid: "697aabe47da4492b87a75d405eb04bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0106846b4b4e4e74a75ba712a907b4e9" is_pre_election: true
I20260926 14:54:02.135350 32763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.131240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.68.193:41123 (local address 127.12.68.194:44835)
0926 14:54:02.131459 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:02.131464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:02.131476 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:54:02.131513 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:02.131516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:02.131586 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:02.131650 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:02.132030 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.132575 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.134047 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.134607 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.134703 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.134747 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:02.135085 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 14:54:02.135194 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260926 14:54:02.136411 32701 raft_consensus.cc:2454] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 697aabe47da4492b87a75d405eb04bdd in term 0.
I20260926 14:54:02.136740 32570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c9126b4fbc48b48bc5c99cfb00ad9d" candidate_uuid: "697aabe47da4492b87a75d405eb04bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d842933ab12d49c58360ac129e5ebeaf" is_pre_election: true
I20260926 14:54:02.136879 32375 leader_election.cc:302] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0106846b4b4e4e74a75ba712a907b4e9, 697aabe47da4492b87a75d405eb04bdd; no voters:
I20260926 14:54:02.137004 32570 raft_consensus.cc:2454] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 697aabe47da4492b87a75d405eb04bdd in term 0.
I20260926 14:54:02.137148 304 raft_consensus.cc:2781] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:54:02.137228 304 raft_consensus.cc:482] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:54:02.137270 304 raft_consensus.cc:3037] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:02.138216 304 raft_consensus.cc:504] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697aabe47da4492b87a75d405eb04bdd" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 39717 } } peers { permanent_uuid: "0106846b4b4e4e74a75ba712a907b4e9" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41637 } } peers { permanent_uuid: "d842933ab12d49c58360ac129e5ebeaf" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 44835 } }
I20260926 14:54:02.138507 304 leader_election.cc:288] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [CANDIDATE]: Term 1 election: Requested vote from peers 0106846b4b4e4e74a75ba712a907b4e9 (127.12.68.195:41637), d842933ab12d49c58360ac129e5ebeaf (127.12.68.194:44835)
I20260926 14:54:02.138815 32701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c9126b4fbc48b48bc5c99cfb00ad9d" candidate_uuid: "697aabe47da4492b87a75d405eb04bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0106846b4b4e4e74a75ba712a907b4e9"
I20260926 14:54:02.138891 32570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c9126b4fbc48b48bc5c99cfb00ad9d" candidate_uuid: "697aabe47da4492b87a75d405eb04bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d842933ab12d49c58360ac129e5ebeaf"
I20260926 14:54:02.138936 32701 raft_consensus.cc:3037] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:02.139034 32570 raft_consensus.cc:3037] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:02.140184 32570 raft_consensus.cc:2454] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 697aabe47da4492b87a75d405eb04bdd in term 1.
I20260926 14:54:02.140184 32701 raft_consensus.cc:2454] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 697aabe47da4492b87a75d405eb04bdd in term 1.
I20260926 14:54:02.140501 32375 leader_election.cc:302] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 697aabe47da4492b87a75d405eb04bdd, d842933ab12d49c58360ac129e5ebeaf; no voters:
I20260926 14:54:02.140640 304 raft_consensus.cc:2781] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:02.140843 304 raft_consensus.cc:686] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [term 1 LEADER]: Becoming Leader. State: Replica: 697aabe47da4492b87a75d405eb04bdd, State: Running, Role: LEADER
I20260926 14:54:02.141149 304 consensus_queue.cc:237] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697aabe47da4492b87a75d405eb04bdd" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 39717 } } peers { permanent_uuid: "0106846b4b4e4e74a75ba712a907b4e9" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41637 } } peers { permanent_uuid: "d842933ab12d49c58360ac129e5ebeaf" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 44835 } }
I20260926 14:54:02.144126 32297 catalog_manager.cc:5495] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 697aabe47da4492b87a75d405eb04bdd (127.12.68.193). New cstate: current_term: 1 leader_uuid: "697aabe47da4492b87a75d405eb04bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697aabe47da4492b87a75d405eb04bdd" member_type: VOTER last_known_addr { host: "127.12.68.193" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0106846b4b4e4e74a75ba712a907b4e9" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d842933ab12d49c58360ac129e5ebeaf" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 44835 } health_report { overall_health: UNKNOWN } } }
I20260926 14:54:02.165130 32760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.161721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53716 (local address 127.12.68.193:39717)
0926 14:54:02.161842 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:02.161851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:02.161864 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:54:02.161946 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:02.161948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:02.162002 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:02.162100 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:02.162327 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.162824 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.163715 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.163994 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.164076 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.164396 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:02.164811 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:02.164900 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:02.164975 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 14:54:02.165021 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:54:02.170820 32701 raft_consensus.cc:1261] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9 [term 1 FOLLOWER]: Refusing update from remote peer 697aabe47da4492b87a75d405eb04bdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:02.170820 32570 raft_consensus.cc:1261] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf [term 1 FOLLOWER]: Refusing update from remote peer 697aabe47da4492b87a75d405eb04bdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:02.171468 306 consensus_queue.cc:1035] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0106846b4b4e4e74a75ba712a907b4e9" member_type: VOTER last_known_addr { host: "127.12.68.195" port: 41637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:02.171644 304 consensus_queue.cc:1035] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d842933ab12d49c58360ac129e5ebeaf" member_type: VOTER last_known_addr { host: "127.12.68.194" port: 44835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:02.181005 309 mvcc.cc:205] Tried to move back new op lower bound from 7333619475125104640 to 7333619475012771840. Current Snapshot: MvccSnapshot[applied={T|T < 7333619475125104640 or (T in {7333619475125104640})}]
I20260926 14:54:02.186035 32296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:54:02.186167 32296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:54:02.188325 32296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8805a85b67704f62b8b24c98cca9ded2: Sending DeleteTablet for 3 replicas of tablet 39c9126b4fbc48b48bc5c99cfb00ad9d
I20260926 14:54:02.189179 32550 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c9126b4fbc48b48bc5c99cfb00ad9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:02 UTC) from {username='slave'} at 127.0.0.1:37504
I20260926 14:54:02.189194 32681 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c9126b4fbc48b48bc5c99cfb00ad9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:02 UTC) from {username='slave'} at 127.0.0.1:46036
I20260926 14:54:02.189647 319 tablet_replica.cc:330] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf: stopping tablet replica
I20260926 14:54:02.189664 32419 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c9126b4fbc48b48bc5c99cfb00ad9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:02 UTC) from {username='slave'} at 127.0.0.1:53704
I20260926 14:54:02.189934 318 tablet_replica.cc:330] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9: stopping tablet replica
I20260926 14:54:02.190065 320 tablet_replica.cc:330] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd: stopping tablet replica
I20260926 14:54:02.190174 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 32357
I20260926 14:54:02.190339 320 raft_consensus.cc:2229] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 697aabe47da4492b87a75d405eb04bdd [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:54:02.190624 319 raft_consensus.cc:2229] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:54:02.190991 319 raft_consensus.cc:2258] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:54:02.191227 318 raft_consensus.cc:2229] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:54:02.191510 318 raft_consensus.cc:2258] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:54:02.192561 319 ts_tablet_manager.cc:1905] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:02.192899 318 ts_tablet_manager.cc:1905] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:02.195231 319 ts_tablet_manager.cc:1918] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:54:02.195238 318 ts_tablet_manager.cc:1918] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:54:02.195330 319 log.cc:1196] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-1/wal/wals/39c9126b4fbc48b48bc5c99cfb00ad9d
I20260926 14:54:02.195330 318 log.cc:1196] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9: Deleting WAL directory at /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434204424021-12563-0/minicluster-data/ts-2/wal/wals/39c9126b4fbc48b48bc5c99cfb00ad9d
I20260926 14:54:02.195616 318 ts_tablet_manager.cc:1939] T 39c9126b4fbc48b48bc5c99cfb00ad9d P 0106846b4b4e4e74a75ba712a907b4e9: Deleting consensus metadata
I20260926 14:54:02.195616 319 ts_tablet_manager.cc:1939] T 39c9126b4fbc48b48bc5c99cfb00ad9d P d842933ab12d49c58360ac129e5ebeaf: Deleting consensus metadata
I20260926 14:54:02.196806 32284 catalog_manager.cc:4841] TS d842933ab12d49c58360ac129e5ebeaf (127.12.68.194:44835): tablet 39c9126b4fbc48b48bc5c99cfb00ad9d (table test-table [id=452dc35060d74fee9119b2af9a7aea89]) successfully deleted
W20260926 14:54:02.196941 32281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.68.193:39717 (error 108)
I20260926 14:54:02.196988 32284 catalog_manager.cc:4841] TS 0106846b4b4e4e74a75ba712a907b4e9 (127.12.68.195:41637): tablet 39c9126b4fbc48b48bc5c99cfb00ad9d (table test-table [id=452dc35060d74fee9119b2af9a7aea89]) successfully deleted
I20260926 14:54:02.197149 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 32488
I20260926 14:54:02.198814 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.198524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.193:39717 (local address 127.0.0.1:53728)
0926 14:54:02.198660 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:02.198734 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.193:39717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 14:54:02.198958 32281 catalog_manager.cc:4584] TS 697aabe47da4492b87a75d405eb04bdd (127.12.68.193:39717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39c9126b4fbc48b48bc5c99cfb00ad9d: Network error: Client connection negotiation failed: client connection to 127.12.68.193:39717: connect: Connection refused (error 111)
I20260926 14:54:02.202517 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 32619
I20260926 14:54:02.206969 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 32266
2026-09-26T14:54:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (922 ms)
[----------] 6 tests from EncryptionPolicyTest (9146 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:02 dist-test-slave-24mn krb5kdc[324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:02 dist-test-slave-24mn krb5kdc[324](info): set up 2 sockets
Sep 26 14:54:02 dist-test-slave-24mn krb5kdc[324](info): commencing operation
krb5kdc: starting...
W20260926 14:54:03.151721 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:03 dist-test-slave-24mn krb5kdc[324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:54:03.262279 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:03.366096 337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:03.366341 337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:03.366400 337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:03.370342 337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:03.370398 337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:03.370424 337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:03.370445 337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:03.370473 337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:03.375525 337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:03.376506 337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:03.377449 337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:03.385352 345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:03.385360 343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:03.385329 342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:03.385350 337 server_base.cc:1034] running on GCE node
I20260926 14:54:03.386113 337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:03.387202 337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:03.388352 337 hybrid_clock.cc:648] HybridClock initialized: now 1790434443388333 us; error 32 us; skew 500 ppm
Sep 26 14:54:03 dist-test-slave-24mn krb5kdc[324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:03.397027 337 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:54:03.397863 337 webserver.cc:460] Webserver started at http://127.0.0.1:41007/ using document root <none> and password file <none>
I20260926 14:54:03.398372 337 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:03.398432 337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:03.398636 337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:03.400645 337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "8f161999b2e44e1fb630582084f68361"
format_stamp: "Formatted at 2026-09-26 14:54:03 on dist-test-slave-24mn"
server_key: "8370b843975b674234deb52ed88ab417"
server_key_iv: "6effac9b6cb054d32390ce2b3e8dc4fa"
server_key_version: "encryptionkey@0"
I20260926 14:54:03.401142 337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "8f161999b2e44e1fb630582084f68361"
format_stamp: "Formatted at 2026-09-26 14:54:03 on dist-test-slave-24mn"
server_key: "8370b843975b674234deb52ed88ab417"
server_key_iv: "6effac9b6cb054d32390ce2b3e8dc4fa"
server_key_version: "encryptionkey@0"
I20260926 14:54:03.404588 337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:54:03.406847 352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:03.407997 337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:54:03.408100 337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "8f161999b2e44e1fb630582084f68361"
format_stamp: "Formatted at 2026-09-26 14:54:03 on dist-test-slave-24mn"
server_key: "8370b843975b674234deb52ed88ab417"
server_key_iv: "6effac9b6cb054d32390ce2b3e8dc4fa"
server_key_version: "encryptionkey@0"
I20260926 14:54:03.408206 337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:03.425042 337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:03.427872 337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:03.428051 337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:03.434415 337 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46825
I20260926 14:54:03.434427 404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46825 every 8 connection(s)
I20260926 14:54:03.435336 337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:03.437407 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 337
I20260926 14:54:03.437532 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:03.437798 12563 external_mini_cluster.cc:1428] Setting key a95a9269bd714d681ef49f04f2a09e3d
I20260926 14:54:03.437994 405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:03.444830 405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361: Bootstrap starting.
Sep 26 14:54:03 dist-test-slave-24mn krb5kdc[324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:54:03.447175 405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:03.448058 405 log.cc:824] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:03.450083 405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361: No bootstrap required, opened a new log
I20260926 14:54:03.452008 408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:03.439039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35549 (local address 127.0.0.1:46825)
0926 14:54:03.439466 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:03.439484 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:03.439524 (+ 40us) server_negotiation.cc:407] Connection header received
0926 14:54:03.440230 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:03.440252 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:03.440696 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:03.441064 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:03.441645 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:03.443102 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:03.443909 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:03.444566 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:03.444724 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:03.447112 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:03.447118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:03.447134 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:03.447164 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:03.448877 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:03.449334 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:03.449340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:03.449345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:03.449472 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:03.449728 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:03.449731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:03.449732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:03.450208 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:03.450410 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:03.450632 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 14:54:03.450840 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":136,"threads_started":1}
I20260926 14:54:03.453804 405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f161999b2e44e1fb630582084f68361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46825 } }
I20260926 14:54:03.454044 405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:03.454103 405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f161999b2e44e1fb630582084f68361, State: Initialized, Role: FOLLOWER
I20260926 14:54:03.454506 405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f161999b2e44e1fb630582084f68361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46825 } }
I20260926 14:54:03.454666 405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:03.454731 405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:03.454772 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:03.454826 405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:03.455871 405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f161999b2e44e1fb630582084f68361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46825 } }
I20260926 14:54:03.456262 405 leader_election.cc:302] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f161999b2e44e1fb630582084f68361; no voters:
I20260926 14:54:03.456574 405 leader_election.cc:288] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:03.456669 414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:03.456893 414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [term 1 LEADER]: Becoming Leader. State: Replica: 8f161999b2e44e1fb630582084f68361, State: Running, Role: LEADER
I20260926 14:54:03.457185 414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f161999b2e44e1fb630582084f68361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46825 } }
I20260926 14:54:03.457715 405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:03.458228 414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f161999b2e44e1fb630582084f68361. Latest consensus state: current_term: 1 leader_uuid: "8f161999b2e44e1fb630582084f68361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f161999b2e44e1fb630582084f68361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46825 } } }
I20260926 14:54:03.458341 414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:03.458720 415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f161999b2e44e1fb630582084f68361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f161999b2e44e1fb630582084f68361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46825 } } }
I20260926 14:54:03.458873 415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:03.458909 422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:03.461990 422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:03.466771 408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:03.457168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58577 (local address 127.0.0.1:46825)
0926 14:54:03.457276 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:03.457280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:03.458706 (+ 1426us) server_negotiation.cc:407] Connection header received
0926 14:54:03.459112 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:03.459115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:03.459155 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:03.459211 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:03.460086 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:03.460861 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:03.461998 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:03.462277 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:03.462364 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:03.463791 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:03.463794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:03.463797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:03.463820 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:03.465101 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:03.465510 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:03.465514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:03.465516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:03.465586 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:03.465890 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:03.465893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:03.465895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:03.466085 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:03.466219 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:03.466551 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 14:54:03.466630 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:54:03.471966 422 catalog_manager.cc:1347] Generated new cluster ID: 600222514b2e4011bcb131d2e2922541
I20260926 14:54:03.472021 422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:03.489023 422 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:03.489895 422 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:03.498467 422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f161999b2e44e1fb630582084f68361: Generated new TSK 0
I20260926 14:54:03.499078 422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:03.504009 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 337
W20260926 14:54:03.508525 322 connection.cc:535] client connection to 127.0.0.1:46825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46825): Connection reset by peer (error 104)
2026-09-26T14:54:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:03 dist-test-slave-24mn krb5kdc[436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:03 dist-test-slave-24mn krb5kdc[436](info): set up 2 sockets
Sep 26 14:54:03 dist-test-slave-24mn krb5kdc[436](info): commencing operation
krb5kdc: starting...
W20260926 14:54:04.439682 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:04 dist-test-slave-24mn krb5kdc[436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:54:04.548986 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44919
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:04.653618 449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:04.653875 449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:04.653915 449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:04.657904 449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:04.657963 449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:04.657981 449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:04.657997 449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:04.658011 449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:04.663193 449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44919
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:04.664212 449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:04.665160 449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:54:04.673125 449 server_base.cc:1034] running on GCE node
W20260926 14:54:04.673065 454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:04.673144 455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:04.673067 457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:04.673852 449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:04.674880 449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:04.676033 449 hybrid_clock.cc:648] HybridClock initialized: now 1790434444676017 us; error 29 us; skew 500 ppm
Sep 26 14:54:04 dist-test-slave-24mn krb5kdc[436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:04.678625 449 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:54:04.679461 449 webserver.cc:460] Webserver started at http://127.0.0.1:41361/ using document root <none> and password file <none>
I20260926 14:54:04.680002 449 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:04.680065 449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:04.680274 449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:04.682214 449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "ee5e94fa37764788bd5a7ae0a98b05c9"
format_stamp: "Formatted at 2026-09-26 14:54:04 on dist-test-slave-24mn"
server_key: "3e0bbe4b4305ced1d56ec425db7295a0"
server_key_iv: "4dbbab9bd124e03f3de1cc0c09c8fefb"
server_key_version: "encryptionkey@0"
I20260926 14:54:04.682734 449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "ee5e94fa37764788bd5a7ae0a98b05c9"
format_stamp: "Formatted at 2026-09-26 14:54:04 on dist-test-slave-24mn"
server_key: "3e0bbe4b4305ced1d56ec425db7295a0"
server_key_iv: "4dbbab9bd124e03f3de1cc0c09c8fefb"
server_key_version: "encryptionkey@0"
I20260926 14:54:04.686251 449 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:54:04.688457 464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:04.689602 449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:54:04.689709 449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "ee5e94fa37764788bd5a7ae0a98b05c9"
format_stamp: "Formatted at 2026-09-26 14:54:04 on dist-test-slave-24mn"
server_key: "3e0bbe4b4305ced1d56ec425db7295a0"
server_key_iv: "4dbbab9bd124e03f3de1cc0c09c8fefb"
server_key_version: "encryptionkey@0"
I20260926 14:54:04.689796 449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:04.699573 449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:04.702149 449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:04.702307 449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:04.708948 449 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44919
I20260926 14:54:04.708957 516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44919 every 8 connection(s)
I20260926 14:54:04.709830 449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:04.712455 517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:04.714030 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 449
I20260926 14:54:04.714169 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:04.714462 12563 external_mini_cluster.cc:1428] Setting key 14219461692fe4fbff44ee0ff158bf8a
I20260926 14:54:04.718528 517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9: Bootstrap starting.
I20260926 14:54:04.720650 517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:04.721407 517 log.cc:824] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:04.723075 520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:04.715806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41225 (local address 127.0.0.1:44919)
0926 14:54:04.716249 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:04.716263 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:04.716297 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:54:04.717015 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:04.717034 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:04.717386 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:04.717688 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:04.718075 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:04.719038 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:04.719942 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:04.720392 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:04.720531 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:04.721375 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:04.721381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:04.721397 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:04.721846 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:04.721908 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:04.721939 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:04.722068 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:04.722255 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 14:54:04.722388 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":120,"threads_started":1}
I20260926 14:54:04.723245 517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9: No bootstrap required, opened a new log
I20260926 14:54:04.725770 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:04.727197 517 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee5e94fa37764788bd5a7ae0a98b05c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44919 } }
I20260926 14:54:04.727464 517 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:04.727522 517 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5e94fa37764788bd5a7ae0a98b05c9, State: Initialized, Role: FOLLOWER
I20260926 14:54:04.727950 517 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee5e94fa37764788bd5a7ae0a98b05c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44919 } }
I20260926 14:54:04.728112 517 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:04.728181 517 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:04.728291 517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:04.729171 517 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee5e94fa37764788bd5a7ae0a98b05c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44919 } }
I20260926 14:54:04.729526 517 leader_election.cc:302] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee5e94fa37764788bd5a7ae0a98b05c9; no voters:
I20260926 14:54:04.729827 517 leader_election.cc:288] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:04.729910 527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:04.730123 527 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [term 1 LEADER]: Becoming Leader. State: Replica: ee5e94fa37764788bd5a7ae0a98b05c9, State: Running, Role: LEADER
I20260926 14:54:04.730459 527 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee5e94fa37764788bd5a7ae0a98b05c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44919 } }
I20260926 14:54:04.731067 517 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:04.731969 528 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee5e94fa37764788bd5a7ae0a98b05c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee5e94fa37764788bd5a7ae0a98b05c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44919 } } }
I20260926 14:54:04.732088 528 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:04.732134 520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:04.727923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36249 (local address 127.0.0.1:44919)
0926 14:54:04.728037 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:04.728040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:04.728112 (+ 72us) server_negotiation.cc:407] Connection header received
0926 14:54:04.728243 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:04.728246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:04.728287 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:04.728356 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:04.728613 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:04.729093 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:04.730190 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:04.730516 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:04.730589 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:04.731581 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:04.731585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:04.731587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:04.731599 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:04.731626 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:04.731639 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:04.731714 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:04.731981 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 14:54:04.732016 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:54:04.732945 529 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee5e94fa37764788bd5a7ae0a98b05c9. Latest consensus state: current_term: 1 leader_uuid: "ee5e94fa37764788bd5a7ae0a98b05c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee5e94fa37764788bd5a7ae0a98b05c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44919 } } }
I20260926 14:54:04.733090 529 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9 [sys.catalog]: This master's current role is: LEADER
W20260926 14:54:04.735865 543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:54:04.735958 543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:54:04.736032 536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:04.736888 536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:04.742445 536 catalog_manager.cc:1347] Generated new cluster ID: 935c73cba8f8447caf4b2dafc5aad7f2
I20260926 14:54:04.742493 536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:04.750924 536 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:04.751833 536 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:04.757043 536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee5e94fa37764788bd5a7ae0a98b05c9: Generated new TSK 0
I20260926 14:54:04.757615 536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:04.761638 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 449
W20260926 14:54:04.766021 434 connection.cc:535] client connection to 127.0.0.1:44919 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44919): Connection reset by peer (error 104)
2026-09-26T14:54:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1258 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:04 dist-test-slave-24mn krb5kdc[548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:04 dist-test-slave-24mn krb5kdc[548](info): set up 2 sockets
Sep 26 14:54:04 dist-test-slave-24mn krb5kdc[548](info): commencing operation
krb5kdc: starting...
W20260926 14:54:05.715709 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:05 dist-test-slave-24mn krb5kdc[548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:54:05.827323 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:05.930110 561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:05.930356 561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:05.930410 561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:05.934340 561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:05.934389 561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:05.934415 561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:05.934437 561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:05.934458 561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:05.939460 561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37105
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:05.940412 561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:05.941314 561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:05.948994 566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:05.949010 567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:05.949002 569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:05.949055 561 server_base.cc:1034] running on GCE node
I20260926 14:54:05.949683 561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:05.950623 561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:05.951771 561 hybrid_clock.cc:648] HybridClock initialized: now 1790434445951757 us; error 27 us; skew 500 ppm
Sep 26 14:54:05 dist-test-slave-24mn krb5kdc[548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:05.954242 561 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:54:05.955019 561 webserver.cc:460] Webserver started at http://127.0.0.1:42657/ using document root <none> and password file <none>
I20260926 14:54:05.955499 561 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:05.955552 561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:05.955719 561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:05.957613 561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "10c76e3951894fe5ba653d80737a780d"
format_stamp: "Formatted at 2026-09-26 14:54:05 on dist-test-slave-24mn"
server_key: "1100c5025540df93a16cddc532fd1267"
server_key_iv: "c9fe603929b742af48e6315750a40cf4"
server_key_version: "encryptionkey@0"
I20260926 14:54:05.958101 561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "10c76e3951894fe5ba653d80737a780d"
format_stamp: "Formatted at 2026-09-26 14:54:05 on dist-test-slave-24mn"
server_key: "1100c5025540df93a16cddc532fd1267"
server_key_iv: "c9fe603929b742af48e6315750a40cf4"
server_key_version: "encryptionkey@0"
I20260926 14:54:05.961498 561 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 14:54:05.963613 576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:05.964754 561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:05.964859 561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "10c76e3951894fe5ba653d80737a780d"
format_stamp: "Formatted at 2026-09-26 14:54:05 on dist-test-slave-24mn"
server_key: "1100c5025540df93a16cddc532fd1267"
server_key_iv: "c9fe603929b742af48e6315750a40cf4"
server_key_version: "encryptionkey@0"
I20260926 14:54:05.964943 561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:05.969069 561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:05.971598 561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:05.971748 561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:05.977890 561 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37105
I20260926 14:54:05.977905 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37105 every 8 connection(s)
I20260926 14:54:05.978683 561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:05.981205 629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:05.981997 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 561
I20260926 14:54:05.982139 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:05.982434 12563 external_mini_cluster.cc:1428] Setting key 3b2aef287f6af5b98b46f7ef18d7384d
I20260926 14:54:05.986631 629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d: Bootstrap starting.
I20260926 14:54:05.987181 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:05.983595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51563 (local address 127.0.0.1:37105)
0926 14:54:05.983988 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:05.984006 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:05.984182 (+ 176us) server_negotiation.cc:407] Connection header received
0926 14:54:05.984852 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:05.984867 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:05.985136 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:05.985413 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:05.986097 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:05.986101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:05.986112 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:05.986122 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:05.986177 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:05.986196 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:05.986310 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:05.986379 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 14:54:05.986516 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":99,"threads_started":1}
I20260926 14:54:05.988647 629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:05.989377 629 log.cc:824] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:05.989614 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:05.991369 629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d: No bootstrap required, opened a new log
I20260926 14:54:05.992988 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:05.991573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50473 (local address 127.0.0.1:37105)
0926 14:54:05.991681 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:05.991685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:05.991828 (+ 143us) server_negotiation.cc:407] Connection header received
0926 14:54:05.991976 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:05.991979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:05.992010 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:05.992095 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:05.992675 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:05.992679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:05.992682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:05.992693 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:05.992718 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:05.992731 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:05.992810 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:05.992853 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 14:54:05.992884 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:54:05.995107 629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c76e3951894fe5ba653d80737a780d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37105 } }
I20260926 14:54:05.995294 629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:05.995325 629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c76e3951894fe5ba653d80737a780d, State: Initialized, Role: FOLLOWER
I20260926 14:54:05.995654 629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c76e3951894fe5ba653d80737a780d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37105 } }
I20260926 14:54:05.995754 629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:05.995795 629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:05.995857 629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:05.996610 629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c76e3951894fe5ba653d80737a780d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37105 } }
I20260926 14:54:05.996869 629 leader_election.cc:302] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10c76e3951894fe5ba653d80737a780d; no voters:
I20260926 14:54:05.997100 629 leader_election.cc:288] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:05.997184 639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:05.997387 639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [term 1 LEADER]: Becoming Leader. State: Replica: 10c76e3951894fe5ba653d80737a780d, State: Running, Role: LEADER
I20260926 14:54:05.997727 639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c76e3951894fe5ba653d80737a780d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37105 } }
I20260926 14:54:05.998137 629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:05.999387 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10c76e3951894fe5ba653d80737a780d. Latest consensus state: current_term: 1 leader_uuid: "10c76e3951894fe5ba653d80737a780d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c76e3951894fe5ba653d80737a780d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37105 } } }
I20260926 14:54:05.999424 640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10c76e3951894fe5ba653d80737a780d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c76e3951894fe5ba653d80737a780d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37105 } } }
I20260926 14:54:05.999560 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:05.999567 640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d [sys.catalog]: This master's current role is: LEADER
W20260926 14:54:06.002517 654 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:54:06.002588 654 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:54:06.002652 650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:06.003358 650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:06.008628 650 catalog_manager.cc:1347] Generated new cluster ID: 7a704a42f4ed4ae1a0e85163dcc71be0
I20260926 14:54:06.008692 650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:06.021910 650 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:06.022226 650 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:06.036932 650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10c76e3951894fe5ba653d80737a780d: Generated new TSK 0
I20260926 14:54:06.037729 650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:06.044816 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 561
2026-09-26T14:54:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:06 dist-test-slave-24mn krb5kdc[660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:06 dist-test-slave-24mn krb5kdc[660](info): set up 2 sockets
Sep 26 14:54:06 dist-test-slave-24mn krb5kdc[660](info): commencing operation
krb5kdc: starting...
W20260926 14:54:06.995749 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:07 dist-test-slave-24mn krb5kdc[660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:54:07.110553 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:07.215320 673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:07.215574 673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:07.215631 673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:07.219518 673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:07.219574 673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:07.219600 673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:07.219621 673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:07.219641 673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:07.225080 673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:07.226135 673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:07.227099 673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:07.235271 678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:07.235330 679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:07.235381 673 server_base.cc:1034] running on GCE node
W20260926 14:54:07.235332 681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:07.236063 673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:07.237075 673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:07.238263 673 hybrid_clock.cc:648] HybridClock initialized: now 1790434447238250 us; error 26 us; skew 500 ppm
Sep 26 14:54:07 dist-test-slave-24mn krb5kdc[660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:07.240935 673 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:54:07.241730 673 webserver.cc:460] Webserver started at http://127.0.0.1:35611/ using document root <none> and password file <none>
I20260926 14:54:07.242214 673 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:07.242266 673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:07.242455 673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:07.244390 673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "4d56fcacd6234833b3393749a924152d"
format_stamp: "Formatted at 2026-09-26 14:54:07 on dist-test-slave-24mn"
server_key: "bb8dd884c8e2fa68a16b1e3a9b300414"
server_key_iv: "6762f0f3e08bd4223090cca9380f1b59"
server_key_version: "encryptionkey@0"
I20260926 14:54:07.244856 673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "4d56fcacd6234833b3393749a924152d"
format_stamp: "Formatted at 2026-09-26 14:54:07 on dist-test-slave-24mn"
server_key: "bb8dd884c8e2fa68a16b1e3a9b300414"
server_key_iv: "6762f0f3e08bd4223090cca9380f1b59"
server_key_version: "encryptionkey@0"
I20260926 14:54:07.248248 673 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:54:07.250463 688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:07.251763 673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:07.251863 673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "4d56fcacd6234833b3393749a924152d"
format_stamp: "Formatted at 2026-09-26 14:54:07 on dist-test-slave-24mn"
server_key: "bb8dd884c8e2fa68a16b1e3a9b300414"
server_key_iv: "6762f0f3e08bd4223090cca9380f1b59"
server_key_version: "encryptionkey@0"
I20260926 14:54:07.251995 673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:07.273993 673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:07.277017 673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:07.277192 673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:07.283710 673 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41101
I20260926 14:54:07.283741 740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41101 every 8 connection(s)
I20260926 14:54:07.284607 673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:07.285974 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 673
I20260926 14:54:07.286083 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:07.286342 12563 external_mini_cluster.cc:1428] Setting key 91a7f2aee2c8d0428b413410b11a2e3e
I20260926 14:54:07.287438 741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:07.293859 741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d: Bootstrap starting.
Sep 26 14:54:07 dist-test-slave-24mn krb5kdc[660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:54:07.296387 741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:07.297281 741 log.cc:824] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:07.299427 741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d: No bootstrap required, opened a new log
I20260926 14:54:07.301386 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:07.287653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48369 (local address 127.0.0.1:41101)
0926 14:54:07.288063 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:07.288081 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:07.288122 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:54:07.288941 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:07.288963 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:07.289399 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:07.289750 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:07.290402 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:07.291839 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:07.292721 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:07.293376 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:07.293538 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:07.296047 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:07.296053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:07.296070 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:07.296101 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:07.298067 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:07.298511 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:07.298516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:07.298522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:07.298661 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:07.298972 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:07.298975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:07.298977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:07.299481 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:07.299664 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:07.299950 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 14:54:07.300249 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":131,"threads_started":1}
I20260926 14:54:07.302850 741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d56fcacd6234833b3393749a924152d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } }
I20260926 14:54:07.303058 741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:07.303138 741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d56fcacd6234833b3393749a924152d, State: Initialized, Role: FOLLOWER
I20260926 14:54:07.303588 741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d56fcacd6234833b3393749a924152d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } }
I20260926 14:54:07.303742 741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:07.303804 741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:07.303921 741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:07.304013 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:07.304765 741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d56fcacd6234833b3393749a924152d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } }
I20260926 14:54:07.305089 741 leader_election.cc:302] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d56fcacd6234833b3393749a924152d; no voters:
I20260926 14:54:07.305495 741 leader_election.cc:288] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:07.305557 750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:07.305751 750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [term 1 LEADER]: Becoming Leader. State: Replica: 4d56fcacd6234833b3393749a924152d, State: Running, Role: LEADER
I20260926 14:54:07.306033 750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d56fcacd6234833b3393749a924152d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } }
I20260926 14:54:07.306733 741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:07.307533 752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d56fcacd6234833b3393749a924152d. Latest consensus state: current_term: 1 leader_uuid: "4d56fcacd6234833b3393749a924152d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d56fcacd6234833b3393749a924152d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } } }
I20260926 14:54:07.307668 752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:07.308182 760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:07.308172 751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d56fcacd6234833b3393749a924152d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d56fcacd6234833b3393749a924152d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } } }
I20260926 14:54:07.308902 751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:07.313680 760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:07.317651 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:07.306024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59685 (local address 127.0.0.1:41101)
0926 14:54:07.306113 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:07.306116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:07.306346 (+ 230us) server_negotiation.cc:407] Connection header received
0926 14:54:07.306437 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:07.306440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:07.306480 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:07.306543 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:07.309458 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:07.310415 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:07.311560 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:07.311938 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:07.312025 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:07.314114 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:07.314118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:07.314120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:07.314145 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:07.315721 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:07.316155 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:07.316158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:07.316160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:07.316244 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:07.316504 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:07.316507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:07.316509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:07.316708 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:07.316823 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:07.317405 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 14:54:07.317511 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:54:07.320691 760 catalog_manager.cc:1347] Generated new cluster ID: 5f4c5656087d4339952488394b2e9842
I20260926 14:54:07.320747 760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:07.354621 760 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:07.355633 760 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:07.363485 760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d56fcacd6234833b3393749a924152d: Generated new TSK 0
I20260926 14:54:07.364172 760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:07.370186 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 673
2026-09-26T14:54:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:07 dist-test-slave-24mn krb5kdc[772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:07 dist-test-slave-24mn krb5kdc[772](info): set up 2 sockets
Sep 26 14:54:07 dist-test-slave-24mn krb5kdc[772](info): commencing operation
krb5kdc: starting...
W20260926 14:54:08.315809 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:08 dist-test-slave-24mn krb5kdc[772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:54:08.429041 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:08.534876 785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:08.535115 785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:08.535154 785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:08.539022 785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:08.539074 785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:08.539093 785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:08.539108 785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:08.539121 785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:08.544173 785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:08.545111 785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:08.546113 785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:08.554085 793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:08.554059 791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:08.554075 790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:08.554199 785 server_base.cc:1034] running on GCE node
I20260926 14:54:08.554919 785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:08.556253 785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:08.557408 785 hybrid_clock.cc:648] HybridClock initialized: now 1790434448557387 us; error 39 us; skew 500 ppm
Sep 26 14:54:08 dist-test-slave-24mn krb5kdc[772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:08.563117 785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:54:08.564072 785 webserver.cc:460] Webserver started at http://127.0.0.1:33615/ using document root <none> and password file <none>
I20260926 14:54:08.564630 785 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:08.564697 785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:08.564881 785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:08.566910 785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "78699a6fc59b49afab79bdcd4f7c377c"
format_stamp: "Formatted at 2026-09-26 14:54:08 on dist-test-slave-24mn"
server_key: "b6cae440408a1e6492f285a76637c62f"
server_key_iv: "9e0bdf0fbef37a04c054bce777269cf6"
server_key_version: "encryptionkey@0"
I20260926 14:54:08.567457 785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "78699a6fc59b49afab79bdcd4f7c377c"
format_stamp: "Formatted at 2026-09-26 14:54:08 on dist-test-slave-24mn"
server_key: "b6cae440408a1e6492f285a76637c62f"
server_key_iv: "9e0bdf0fbef37a04c054bce777269cf6"
server_key_version: "encryptionkey@0"
I20260926 14:54:08.571048 785 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:54:08.573537 800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:08.574877 785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:54:08.574995 785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "78699a6fc59b49afab79bdcd4f7c377c"
format_stamp: "Formatted at 2026-09-26 14:54:08 on dist-test-slave-24mn"
server_key: "b6cae440408a1e6492f285a76637c62f"
server_key_iv: "9e0bdf0fbef37a04c054bce777269cf6"
server_key_version: "encryptionkey@0"
I20260926 14:54:08.575093 785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:08.584302 785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:08.586859 785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:08.587023 785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:08.593511 785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44495
I20260926 14:54:08.593523 852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44495 every 8 connection(s)
I20260926 14:54:08.594384 785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:08.596959 853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:08.602182 853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c: Bootstrap starting.
I20260926 14:54:08.604362 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 785
I20260926 14:54:08.604496 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:08.604660 853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:08.604820 12563 external_mini_cluster.cc:1428] Setting key 9ce0ce6a6aa0344eb8d8af8d4c1dec05
I20260926 14:54:08.605436 853 log.cc:824] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:08.607309 853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c: No bootstrap required, opened a new log
I20260926 14:54:08.610741 853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78699a6fc59b49afab79bdcd4f7c377c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } }
I20260926 14:54:08.611008 853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:08.611099 853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78699a6fc59b49afab79bdcd4f7c377c, State: Initialized, Role: FOLLOWER
I20260926 14:54:08.611532 853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78699a6fc59b49afab79bdcd4f7c377c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } }
I20260926 14:54:08.611663 853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:08.611727 853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:08.611821 853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:08.612752 853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78699a6fc59b49afab79bdcd4f7c377c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } }
I20260926 14:54:08.612947 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:08.605907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35669 (local address 127.0.0.1:44495)
0926 14:54:08.606263 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:08.606278 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:08.606311 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:54:08.606924 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:08.606940 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:08.607265 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:08.607586 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:08.608021 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:08.608966 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:08.609832 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:08.610289 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:08.610435 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:08.611324 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:08.611328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:08.611341 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:08.611729 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:08.611799 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:08.611827 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:08.611936 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:08.612046 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 14:54:08.612249 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":74,"threads_started":1}
I20260926 14:54:08.613101 853 leader_election.cc:302] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78699a6fc59b49afab79bdcd4f7c377c; no voters:
I20260926 14:54:08.613427 853 leader_election.cc:288] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:08.613550 858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:08.613749 858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [term 1 LEADER]: Becoming Leader. State: Replica: 78699a6fc59b49afab79bdcd4f7c377c, State: Running, Role: LEADER
I20260926 14:54:08.614046 858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78699a6fc59b49afab79bdcd4f7c377c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } }
I20260926 14:54:08.614661 853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:08.615484 860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78699a6fc59b49afab79bdcd4f7c377c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78699a6fc59b49afab79bdcd4f7c377c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } } }
I20260926 14:54:08.615646 860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:08.615795 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:08.616138 867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:08.616797 859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78699a6fc59b49afab79bdcd4f7c377c. Latest consensus state: current_term: 1 leader_uuid: "78699a6fc59b49afab79bdcd4f7c377c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78699a6fc59b49afab79bdcd4f7c377c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } } }
I20260926 14:54:08.616887 859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:08.619297 867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:08.621726 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:08.617906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37121 (local address 127.0.0.1:44495)
0926 14:54:08.618001 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:08.618005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:08.618202 (+ 197us) server_negotiation.cc:407] Connection header received
0926 14:54:08.618322 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:08.618325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:08.618365 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:08.618430 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:08.618696 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:08.619200 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:08.620249 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:08.620532 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:08.620624 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:08.621352 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:08.621356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:08.621358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:08.621370 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:08.621397 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:08.621407 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:08.621463 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:08.621553 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 14:54:08.621601 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:54:08.625165 867 catalog_manager.cc:1347] Generated new cluster ID: 6907a9130b6f4c66a73317414659089a
I20260926 14:54:08.625233 867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:08.643656 867 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:08.644968 867 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:08.652263 867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78699a6fc59b49afab79bdcd4f7c377c: Generated new TSK 0
I20260926 14:54:08.652917 867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:08.664564 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 785
2026-09-26T14:54:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:08 dist-test-slave-24mn krb5kdc[884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:08 dist-test-slave-24mn krb5kdc[884](info): set up 2 sockets
Sep 26 14:54:08 dist-test-slave-24mn krb5kdc[884](info): commencing operation
krb5kdc: starting...
W20260926 14:54:09.615800 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:09 dist-test-slave-24mn krb5kdc[884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:54:09.726810 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32847
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:09.831290 897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:09.831528 897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:09.831568 897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:09.835446 897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:09.835500 897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:09.835518 897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:09.835532 897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:09.835546 897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:09.840703 897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:09.841697 897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:09.842645 897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:09.850703 902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:09.850817 903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:09.850718 905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:09.850917 897 server_base.cc:1034] running on GCE node
I20260926 14:54:09.851542 897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:09.852552 897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:09.853713 897 hybrid_clock.cc:648] HybridClock initialized: now 1790434449853691 us; error 35 us; skew 500 ppm
Sep 26 14:54:09 dist-test-slave-24mn krb5kdc[884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:09.856261 897 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:54:09.857079 897 webserver.cc:460] Webserver started at http://127.0.0.1:40581/ using document root <none> and password file <none>
I20260926 14:54:09.857589 897 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:09.857650 897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:09.857831 897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:09.859724 897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "327b7ef8d16c40f29d788b92f9f8a5d1"
format_stamp: "Formatted at 2026-09-26 14:54:09 on dist-test-slave-24mn"
server_key: "81339fc1e4deae5f011fa95a43116a00"
server_key_iv: "0e84a4764bef426febe1c7cf5ece5030"
server_key_version: "encryptionkey@0"
I20260926 14:54:09.860278 897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "327b7ef8d16c40f29d788b92f9f8a5d1"
format_stamp: "Formatted at 2026-09-26 14:54:09 on dist-test-slave-24mn"
server_key: "81339fc1e4deae5f011fa95a43116a00"
server_key_iv: "0e84a4764bef426febe1c7cf5ece5030"
server_key_version: "encryptionkey@0"
I20260926 14:54:09.863610 897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:54:09.865928 912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:09.867156 897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:09.867281 897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "327b7ef8d16c40f29d788b92f9f8a5d1"
format_stamp: "Formatted at 2026-09-26 14:54:09 on dist-test-slave-24mn"
server_key: "81339fc1e4deae5f011fa95a43116a00"
server_key_iv: "0e84a4764bef426febe1c7cf5ece5030"
server_key_version: "encryptionkey@0"
I20260926 14:54:09.867380 897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:09.871729 897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:09.874648 897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:09.874809 897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:09.881184 897 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32847
I20260926 14:54:09.881234 964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32847 every 8 connection(s)
I20260926 14:54:09.882035 897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:09.884694 965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:09.889791 965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1: Bootstrap starting.
I20260926 14:54:09.892026 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 897
I20260926 14:54:09.892163 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:09.892196 965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:09.892475 12563 external_mini_cluster.cc:1428] Setting key ab19b5ebcef484752b358370693b402a
I20260926 14:54:09.892994 965 log.cc:824] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:09.895151 965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1: No bootstrap required, opened a new log
I20260926 14:54:09.897444 969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:09.893504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53959 (local address 127.0.0.1:32847)
0926 14:54:09.893933 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:09.893952 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:09.893994 (+ 42us) server_negotiation.cc:407] Connection header received
0926 14:54:09.894805 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:09.894828 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:09.895155 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:09.895520 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:09.896144 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:09.896149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:09.896162 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:09.896172 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:09.896226 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:09.896249 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:09.896360 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:09.896432 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 14:54:09.896575 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":109,"threads_started":1}
I20260926 14:54:09.899129 965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327b7ef8d16c40f29d788b92f9f8a5d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32847 } }
I20260926 14:54:09.899376 965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:09.899441 965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327b7ef8d16c40f29d788b92f9f8a5d1, State: Initialized, Role: FOLLOWER
I20260926 14:54:09.899850 965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327b7ef8d16c40f29d788b92f9f8a5d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32847 } }
I20260926 14:54:09.900015 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:09.900030 965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:09.900087 965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:09.900174 965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:09.901242 965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327b7ef8d16c40f29d788b92f9f8a5d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32847 } }
I20260926 14:54:09.901605 965 leader_election.cc:302] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 327b7ef8d16c40f29d788b92f9f8a5d1; no voters:
I20260926 14:54:09.901922 965 leader_election.cc:288] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:09.901994 974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:09.902220 974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [term 1 LEADER]: Becoming Leader. State: Replica: 327b7ef8d16c40f29d788b92f9f8a5d1, State: Running, Role: LEADER
I20260926 14:54:09.902588 974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327b7ef8d16c40f29d788b92f9f8a5d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32847 } }
I20260926 14:54:09.903241 965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:09.904032 977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "327b7ef8d16c40f29d788b92f9f8a5d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327b7ef8d16c40f29d788b92f9f8a5d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32847 } } }
I20260926 14:54:09.904210 977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:09.904547 975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 327b7ef8d16c40f29d788b92f9f8a5d1. Latest consensus state: current_term: 1 leader_uuid: "327b7ef8d16c40f29d788b92f9f8a5d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327b7ef8d16c40f29d788b92f9f8a5d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32847 } } }
I20260926 14:54:09.904690 975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:09.904886 984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:09.905473 969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:09.902221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45201 (local address 127.0.0.1:32847)
0926 14:54:09.902318 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:09.902322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:09.903758 (+ 1436us) server_negotiation.cc:407] Connection header received
0926 14:54:09.903969 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:09.903973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:09.904006 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:09.904052 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:09.904854 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:09.904858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:09.904860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:09.904873 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:09.904901 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:09.904914 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:09.904995 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:09.905322 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 14:54:09.905361 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:54:09.908255 984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:09.913125 984 catalog_manager.cc:1347] Generated new cluster ID: 5f0d6dd5e20f4c9e8858aa99df60570b
I20260926 14:54:09.913182 984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:09.932533 984 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:09.932883 984 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:09.942056 984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 327b7ef8d16c40f29d788b92f9f8a5d1: Generated new TSK 0
I20260926 14:54:09.942873 984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:09.950189 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 897
2026-09-26T14:54:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:10 dist-test-slave-24mn krb5kdc[996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:10 dist-test-slave-24mn krb5kdc[996](info): set up 2 sockets
Sep 26 14:54:10 dist-test-slave-24mn krb5kdc[996](info): commencing operation
krb5kdc: starting...
W20260926 14:54:10.903702 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:10 dist-test-slave-24mn krb5kdc[996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:54:11.016889 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:11.123170 1009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:11.123409 1009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:11.123466 1009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:11.127339 1009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:11.127393 1009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:11.127419 1009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:11.127441 1009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:11.127471 1009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:11.132501 1009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:11.133423 1009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:11.134347 1009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:11.142385 1015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:11.142477 1014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:11.142520 1009 server_base.cc:1034] running on GCE node
W20260926 14:54:11.142388 1017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:11.143134 1009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:11.144220 1009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:11.145378 1009 hybrid_clock.cc:648] HybridClock initialized: now 1790434451145363 us; error 29 us; skew 500 ppm
Sep 26 14:54:11 dist-test-slave-24mn krb5kdc[996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:11.147996 1009 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:54:11.148772 1009 webserver.cc:460] Webserver started at http://127.0.0.1:33705/ using document root <none> and password file <none>
I20260926 14:54:11.149250 1009 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:11.149304 1009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:11.149457 1009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:11.151448 1009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "e1e492262ec6410bbd3b344ba2f96101"
format_stamp: "Formatted at 2026-09-26 14:54:11 on dist-test-slave-24mn"
server_key: "825c7f58ea84818b175035f5cc63e893"
server_key_iv: "28ea3e2ac788436b927a2d7e03865a13"
server_key_version: "encryptionkey@0"
I20260926 14:54:11.151971 1009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "e1e492262ec6410bbd3b344ba2f96101"
format_stamp: "Formatted at 2026-09-26 14:54:11 on dist-test-slave-24mn"
server_key: "825c7f58ea84818b175035f5cc63e893"
server_key_iv: "28ea3e2ac788436b927a2d7e03865a13"
server_key_version: "encryptionkey@0"
I20260926 14:54:11.155287 1009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:54:11.157485 1024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:11.158602 1009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 14:54:11.158710 1009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "e1e492262ec6410bbd3b344ba2f96101"
format_stamp: "Formatted at 2026-09-26 14:54:11 on dist-test-slave-24mn"
server_key: "825c7f58ea84818b175035f5cc63e893"
server_key_iv: "28ea3e2ac788436b927a2d7e03865a13"
server_key_version: "encryptionkey@0"
I20260926 14:54:11.158804 1009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:11.183736 1009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:11.186419 1009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:11.186583 1009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:11.192862 1009 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45497
I20260926 14:54:11.192894 1076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45497 every 8 connection(s)
I20260926 14:54:11.193737 1009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:11.196276 1077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:11.201418 1077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101: Bootstrap starting.
I20260926 14:54:11.202342 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 1009
I20260926 14:54:11.202450 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:11.202697 12563 external_mini_cluster.cc:1428] Setting key a8765572c0aeaba13d7a1fdfe649c2b9
I20260926 14:54:11.203975 1077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:11.204807 1077 log.cc:824] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:11.206822 1077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101: No bootstrap required, opened a new log
I20260926 14:54:11.210527 1077 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e492262ec6410bbd3b344ba2f96101" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260926 14:54:11.210722 1077 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:11.210754 1077 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e492262ec6410bbd3b344ba2f96101, State: Initialized, Role: FOLLOWER
I20260926 14:54:11.211143 1077 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e492262ec6410bbd3b344ba2f96101" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260926 14:54:11.211254 1077 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 26 14:54:11 dist-test-slave-24mn krb5kdc[996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:54:11.211316 1077 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:11.211412 1077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:11.212343 1077 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e492262ec6410bbd3b344ba2f96101" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260926 14:54:11.212692 1077 leader_election.cc:302] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1e492262ec6410bbd3b344ba2f96101; no voters:
I20260926 14:54:11.212991 1077 leader_election.cc:288] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:11.213058 1082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:11.213256 1082 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [term 1 LEADER]: Becoming Leader. State: Replica: e1e492262ec6410bbd3b344ba2f96101, State: Running, Role: LEADER
I20260926 14:54:11.213531 1082 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e492262ec6410bbd3b344ba2f96101" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260926 14:54:11.214171 1077 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:11.214666 1084 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1e492262ec6410bbd3b344ba2f96101. Latest consensus state: current_term: 1 leader_uuid: "e1e492262ec6410bbd3b344ba2f96101" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e492262ec6410bbd3b344ba2f96101" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } } }
I20260926 14:54:11.214780 1084 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:11.215142 1087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:11.216778 1083 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1e492262ec6410bbd3b344ba2f96101" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e492262ec6410bbd3b344ba2f96101" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } } }
I20260926 14:54:11.216878 1083 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:11.217536 1080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:11.203699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:48011 (local address 127.0.0.1:45497)
0926 14:54:11.204146 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:11.204165 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:11.204208 (+ 43us) server_negotiation.cc:407] Connection header received
0926 14:54:11.205008 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:11.205030 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:11.205401 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:11.205719 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:11.206350 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:11.207766 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:11.208628 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:11.209363 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:11.209525 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:11.211910 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:11.211917 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:11.211933 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:11.211961 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:11.213749 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:11.214306 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:11.214311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:11.214317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:11.214449 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:11.214858 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:11.214861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:11.214863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:11.215365 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:11.215575 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:11.216374 (+ 799us) server_negotiation.cc:299] Negotiation successful
0926 14:54:11.216600 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":138,"threads_started":1}
I20260926 14:54:11.218573 1087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:11.220623 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:11.224792 1087 catalog_manager.cc:1347] Generated new cluster ID: db05d94a27e04a8d8ac8c0182aeb551f
I20260926 14:54:11.224853 1087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:11.231078 1080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:11.222382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:43609 (local address 127.0.0.1:45497)
0926 14:54:11.222474 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:11.222478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:11.222945 (+ 467us) server_negotiation.cc:407] Connection header received
0926 14:54:11.223052 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:11.223055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:11.223102 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:11.223191 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:11.223974 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:11.224813 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:11.225960 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:11.226295 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:11.226384 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:11.227973 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:11.227976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:11.227979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:11.228001 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:11.229468 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:11.229953 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:11.229955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:11.229958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:11.230029 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:11.230314 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:11.230316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:11.230318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:11.230488 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:11.230606 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:11.230855 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 14:54:11.230941 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 14:54:11.252547 1087 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:11.253418 1087 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:11.261884 1087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1e492262ec6410bbd3b344ba2f96101: Generated new TSK 0
I20260926 14:54:11.262560 1087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:11.266695 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 1009
W20260926 14:54:11.271420 994 connection.cc:535] client connection to 127.0.0.1:45497 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45497): Connection reset by peer (error 104)
2026-09-26T14:54:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:11 dist-test-slave-24mn krb5kdc[1108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:11 dist-test-slave-24mn krb5kdc[1108](info): set up 2 sockets
Sep 26 14:54:11 dist-test-slave-24mn krb5kdc[1108](info): commencing operation
krb5kdc: starting...
W20260926 14:54:12.212296 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:12 dist-test-slave-24mn krb5kdc[1108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:54:12.324690 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:12.427475 1121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:12.427718 1121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:12.427775 1121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:12.431702 1121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:12.431758 1121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:12.431784 1121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:12.431805 1121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:12.431826 1121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:12.436928 1121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:12.437862 1121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:12.438788 1121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:12.446455 1127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:12.446483 1129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:12.446470 1126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:12.446700 1121 server_base.cc:1034] running on GCE node
I20260926 14:54:12.447286 1121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:12.448336 1121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:12.449487 1121 hybrid_clock.cc:648] HybridClock initialized: now 1790434452449470 us; error 28 us; skew 500 ppm
Sep 26 14:54:12 dist-test-slave-24mn krb5kdc[1108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:12.452013 1121 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:54:12.452762 1121 webserver.cc:460] Webserver started at http://127.0.0.1:45877/ using document root <none> and password file <none>
I20260926 14:54:12.453279 1121 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:12.453338 1121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:12.453497 1121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:12.455335 1121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "82df84dbfe0c4e2a82833b40bfff7c39"
format_stamp: "Formatted at 2026-09-26 14:54:12 on dist-test-slave-24mn"
server_key: "892588cad713d27def8b10f992e3445c"
server_key_iv: "8d0cf270872e2e48f4e63540806dcba0"
server_key_version: "encryptionkey@0"
I20260926 14:54:12.455817 1121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "82df84dbfe0c4e2a82833b40bfff7c39"
format_stamp: "Formatted at 2026-09-26 14:54:12 on dist-test-slave-24mn"
server_key: "892588cad713d27def8b10f992e3445c"
server_key_iv: "8d0cf270872e2e48f4e63540806dcba0"
server_key_version: "encryptionkey@0"
I20260926 14:54:12.459126 1121 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:54:12.461282 1136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:12.462469 1121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:54:12.462577 1121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "82df84dbfe0c4e2a82833b40bfff7c39"
format_stamp: "Formatted at 2026-09-26 14:54:12 on dist-test-slave-24mn"
server_key: "892588cad713d27def8b10f992e3445c"
server_key_iv: "8d0cf270872e2e48f4e63540806dcba0"
server_key_version: "encryptionkey@0"
I20260926 14:54:12.462656 1121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:12.474581 1121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:12.477159 1121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:12.477299 1121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:12.483325 1121 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38705
I20260926 14:54:12.483345 1188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38705 every 8 connection(s)
I20260926 14:54:12.484195 1121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:12.486781 1189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:12.489765 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 1121
I20260926 14:54:12.489902 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:12.490202 12563 external_mini_cluster.cc:1428] Setting key a30fa2e0fd39f857c5a13ad3b8c96e76
I20260926 14:54:12.492404 1189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39: Bootstrap starting.
I20260926 14:54:12.494444 1189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:12.495114 1189 log.cc:824] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:12.496870 1189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39: No bootstrap required, opened a new log
I20260926 14:54:12.498916 1192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:12.491486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:46277 (local address 127.0.0.1:38705)
0926 14:54:12.491924 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:12.491943 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:12.491978 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:54:12.492605 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:12.492622 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:12.492952 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:12.493232 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:12.493608 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:12.494561 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:12.495443 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:12.495928 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:12.496078 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:12.496883 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:12.496888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:12.496899 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:12.497351 (+ 452us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:12.497413 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:12.497442 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:12.497855 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:12.497947 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 14:54:12.498075 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
I20260926 14:54:12.500080 1189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82df84dbfe0c4e2a82833b40bfff7c39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38705 } }
I20260926 14:54:12.500294 1189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:12.500367 1189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82df84dbfe0c4e2a82833b40bfff7c39, State: Initialized, Role: FOLLOWER
I20260926 14:54:12.500797 1189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82df84dbfe0c4e2a82833b40bfff7c39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38705 } }
I20260926 14:54:12.500949 1189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:12.501009 1189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:12.501098 1189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:12.501755 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:12.501893 1189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82df84dbfe0c4e2a82833b40bfff7c39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38705 } }
I20260926 14:54:12.502188 1189 leader_election.cc:302] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82df84dbfe0c4e2a82833b40bfff7c39; no voters:
I20260926 14:54:12.502449 1189 leader_election.cc:288] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:12.502525 1194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:12.502682 1194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [term 1 LEADER]: Becoming Leader. State: Replica: 82df84dbfe0c4e2a82833b40bfff7c39, State: Running, Role: LEADER
I20260926 14:54:12.502996 1194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82df84dbfe0c4e2a82833b40bfff7c39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38705 } }
I20260926 14:54:12.503628 1189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:12.504141 1194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82df84dbfe0c4e2a82833b40bfff7c39. Latest consensus state: current_term: 1 leader_uuid: "82df84dbfe0c4e2a82833b40bfff7c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82df84dbfe0c4e2a82833b40bfff7c39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38705 } } }
I20260926 14:54:12.504245 1194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:12.504508 1195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82df84dbfe0c4e2a82833b40bfff7c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82df84dbfe0c4e2a82833b40bfff7c39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38705 } } }
I20260926 14:54:12.504601 1195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:12.504662 1204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:12.507927 1204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:12.508587 1192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:12.503700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:49431 (local address 127.0.0.1:38705)
0926 14:54:12.503794 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:12.503798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:12.504121 (+ 323us) server_negotiation.cc:407] Connection header received
0926 14:54:12.504297 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:12.504300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:12.504356 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:12.504405 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:12.504794 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:12.505461 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:12.506559 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:12.506893 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:12.506985 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:12.507899 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:12.507903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:12.507906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:12.507918 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:12.507943 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:12.507956 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:12.508045 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:12.508427 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 14:54:12.508462 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:54:12.513103 1204 catalog_manager.cc:1347] Generated new cluster ID: c77b851e5c9b408ea88dee8547bc95b1
I20260926 14:54:12.513163 1204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:12.528479 1204 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:12.529322 1204 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:12.548578 1204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82df84dbfe0c4e2a82833b40bfff7c39: Generated new TSK 0
I20260926 14:54:12.549256 1204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:12.552798 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 1121
W20260926 14:54:12.557313 1106 connection.cc:535] client connection to 127.0.0.1:38705 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38705): Connection reset by peer (error 104)
2026-09-26T14:54:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:12 dist-test-slave-24mn krb5kdc[1220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:12 dist-test-slave-24mn krb5kdc[1220](info): set up 2 sockets
Sep 26 14:54:12 dist-test-slave-24mn krb5kdc[1220](info): commencing operation
krb5kdc: starting...
W20260926 14:54:13.503793 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:13 dist-test-slave-24mn krb5kdc[1220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:54:13.615068 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:13.723152 1233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:13.723412 1233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:13.723453 1233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:13.727339 1233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:13.727391 1233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:13.727408 1233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:13.727422 1233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:13.727437 1233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:13.732502 1233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:13.733454 1233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:13.734388 1233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:13.742882 1241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:13.742940 1238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:13.743005 1233 server_base.cc:1034] running on GCE node
W20260926 14:54:13.742949 1239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:13.743741 1233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:13.744854 1233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:13.746029 1233 hybrid_clock.cc:648] HybridClock initialized: now 1790434453745986 us; error 55 us; skew 500 ppm
Sep 26 14:54:13 dist-test-slave-24mn krb5kdc[1220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:13.754281 1233 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:54:13.755221 1233 webserver.cc:460] Webserver started at http://127.0.0.1:38231/ using document root <none> and password file <none>
I20260926 14:54:13.755761 1233 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:13.755915 1233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:13.756120 1233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:13.758091 1233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "cb1b76e4241741edb178db18fc4252ce"
format_stamp: "Formatted at 2026-09-26 14:54:13 on dist-test-slave-24mn"
server_key: "f0eaa7aee9934a5a4bdf5d32a84e19b4"
server_key_iv: "5dc020d05982eff9aba67dd28495f5be"
server_key_version: "encryptionkey@0"
I20260926 14:54:13.758587 1233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "cb1b76e4241741edb178db18fc4252ce"
format_stamp: "Formatted at 2026-09-26 14:54:13 on dist-test-slave-24mn"
server_key: "f0eaa7aee9934a5a4bdf5d32a84e19b4"
server_key_iv: "5dc020d05982eff9aba67dd28495f5be"
server_key_version: "encryptionkey@0"
I20260926 14:54:13.762094 1233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:54:13.764371 1248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:13.765653 1233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:54:13.765759 1233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "cb1b76e4241741edb178db18fc4252ce"
format_stamp: "Formatted at 2026-09-26 14:54:13 on dist-test-slave-24mn"
server_key: "f0eaa7aee9934a5a4bdf5d32a84e19b4"
server_key_iv: "5dc020d05982eff9aba67dd28495f5be"
server_key_version: "encryptionkey@0"
I20260926 14:54:13.765869 1233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:13.770390 1233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:13.773263 1233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:13.773424 1233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:13.779846 1233 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33739
I20260926 14:54:13.779861 1300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33739 every 8 connection(s)
I20260926 14:54:13.780802 1233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:13.783393 1301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:13.788676 1301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce: Bootstrap starting.
I20260926 14:54:13.790582 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 1233
I20260926 14:54:13.790714 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:13.790992 12563 external_mini_cluster.cc:1428] Setting key dac08d84c3b9607061f577188264339e
I20260926 14:54:13.791236 1301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:13.792105 1301 log.cc:824] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:13.793912 1301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce: No bootstrap required, opened a new log
I20260926 14:54:13.795770 1304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:13.792104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:37225 (local address 127.0.0.1:33739)
0926 14:54:13.792548 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:13.792563 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:13.792596 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:54:13.793272 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:13.793290 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:13.793558 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:13.793835 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:13.794624 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:13.794628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:13.794649 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:13.794659 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:13.794708 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:13.794727 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:13.794808 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:13.794934 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 14:54:13.795074 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":106,"threads_started":1}
I20260926 14:54:13.797430 1301 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1b76e4241741edb178db18fc4252ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } }
I20260926 14:54:13.797739 1301 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:13.797785 1301 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb1b76e4241741edb178db18fc4252ce, State: Initialized, Role: FOLLOWER
I20260926 14:54:13.798245 1301 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1b76e4241741edb178db18fc4252ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } }
I20260926 14:54:13.798390 1301 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:13.798455 1301 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:13.798552 1301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:13.798597 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:13.799391 1301 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1b76e4241741edb178db18fc4252ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } }
I20260926 14:54:13.799692 1301 leader_election.cc:302] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb1b76e4241741edb178db18fc4252ce; no voters:
I20260926 14:54:13.800067 1301 leader_election.cc:288] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:13.800113 1310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:13.800289 1310 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [term 1 LEADER]: Becoming Leader. State: Replica: cb1b76e4241741edb178db18fc4252ce, State: Running, Role: LEADER
I20260926 14:54:13.800616 1310 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1b76e4241741edb178db18fc4252ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } }
I20260926 14:54:13.801275 1301 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:13.802234 1312 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb1b76e4241741edb178db18fc4252ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1b76e4241741edb178db18fc4252ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } } }
I20260926 14:54:13.802520 1312 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:13.802551 1304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:13.800725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:43821 (local address 127.0.0.1:33739)
0926 14:54:13.800817 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:13.800820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:13.801026 (+ 206us) server_negotiation.cc:407] Connection header received
0926 14:54:13.801123 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:13.801126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:13.801155 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:13.801221 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:13.801908 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:13.801912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:13.801914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:13.801926 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:13.801954 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:13.801966 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:13.802353 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:13.802394 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 14:54:13.802431 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:54:13.802430 1311 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb1b76e4241741edb178db18fc4252ce. Latest consensus state: current_term: 1 leader_uuid: "cb1b76e4241741edb178db18fc4252ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1b76e4241741edb178db18fc4252ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } } }
I20260926 14:54:13.802769 1311 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:13.803544 1321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:13.806209 1321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:13.811205 1321 catalog_manager.cc:1347] Generated new cluster ID: 7fa1f68b7052487cb8c55fa3dfdde49b
I20260926 14:54:13.811487 1321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:13.843844 1321 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:13.844221 1321 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:13.849192 1321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb1b76e4241741edb178db18fc4252ce: Generated new TSK 0
I20260926 14:54:13.849948 1321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:13.856221 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 1233
2026-09-26T14:54:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:13 dist-test-slave-24mn krb5kdc[1332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:13 dist-test-slave-24mn krb5kdc[1332](info): set up 2 sockets
Sep 26 14:54:13 dist-test-slave-24mn krb5kdc[1332](info): commencing operation
krb5kdc: starting...
W20260926 14:54:14.811842 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:14 dist-test-slave-24mn krb5kdc[1332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:54:14.938799 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:15.045573 1345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:15.045818 1345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:15.045862 1345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:15.050159 1345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:15.050228 1345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:15.050247 1345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:15.050262 1345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:15.050276 1345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:15.055583 1345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:15.056622 1345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:15.057533 1345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:15.065735 1350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:15.065747 1351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:15.065748 1353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:15.065795 1345 server_base.cc:1034] running on GCE node
I20260926 14:54:15.066461 1345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:15.067538 1345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:15.068693 1345 hybrid_clock.cc:648] HybridClock initialized: now 1790434455068674 us; error 31 us; skew 500 ppm
Sep 26 14:54:15 dist-test-slave-24mn krb5kdc[1332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:15.071346 1345 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:54:15.072177 1345 webserver.cc:460] Webserver started at http://127.0.0.1:36919/ using document root <none> and password file <none>
I20260926 14:54:15.072680 1345 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:15.072741 1345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:15.072918 1345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:15.074883 1345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "515aa88bee2e4b84b418f5704e2da830"
format_stamp: "Formatted at 2026-09-26 14:54:15 on dist-test-slave-24mn"
server_key: "d64e79082c963c37c07ef068a3f76e40"
server_key_iv: "435d022004ff7b8e229b1bc881e72ff0"
server_key_version: "encryptionkey@0"
I20260926 14:54:15.075420 1345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "515aa88bee2e4b84b418f5704e2da830"
format_stamp: "Formatted at 2026-09-26 14:54:15 on dist-test-slave-24mn"
server_key: "d64e79082c963c37c07ef068a3f76e40"
server_key_iv: "435d022004ff7b8e229b1bc881e72ff0"
server_key_version: "encryptionkey@0"
I20260926 14:54:15.078990 1345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:54:15.081344 1360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:15.082481 1345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:54:15.082604 1345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "515aa88bee2e4b84b418f5704e2da830"
format_stamp: "Formatted at 2026-09-26 14:54:15 on dist-test-slave-24mn"
server_key: "d64e79082c963c37c07ef068a3f76e40"
server_key_iv: "435d022004ff7b8e229b1bc881e72ff0"
server_key_version: "encryptionkey@0"
I20260926 14:54:15.082696 1345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:15.094738 1345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:15.097259 1345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:15.097429 1345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:15.103732 1345 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44135
I20260926 14:54:15.103780 1412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44135 every 8 connection(s)
I20260926 14:54:15.104651 1345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:15.107296 1413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:15.112624 1413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830: Bootstrap starting.
I20260926 14:54:15.114429 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 1345
I20260926 14:54:15.114554 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:15.114850 12563 external_mini_cluster.cc:1428] Setting key fc64532206bc161dea54da4289dd446a
I20260926 14:54:15.115252 1413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:15.116043 1413 log.cc:824] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:15.117816 1413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830: No bootstrap required, opened a new log
I20260926 14:54:15.121112 1413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515aa88bee2e4b84b418f5704e2da830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260926 14:54:15.121316 1413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:15.121374 1413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515aa88bee2e4b84b418f5704e2da830, State: Initialized, Role: FOLLOWER
I20260926 14:54:15.121829 1413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515aa88bee2e4b84b418f5704e2da830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260926 14:54:15.121954 1413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:15.122011 1413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:15.122105 1413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:15.123049 1413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515aa88bee2e4b84b418f5704e2da830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
Sep 26 14:54:15 dist-test-slave-24mn krb5kdc[1332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:54:15.123407 1413 leader_election.cc:302] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 515aa88bee2e4b84b418f5704e2da830; no voters:
I20260926 14:54:15.123677 1413 leader_election.cc:288] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:15.123776 1418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:15.124003 1418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [term 1 LEADER]: Becoming Leader. State: Replica: 515aa88bee2e4b84b418f5704e2da830, State: Running, Role: LEADER
I20260926 14:54:15.124406 1418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515aa88bee2e4b84b418f5704e2da830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260926 14:54:15.124922 1413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:15.125474 1418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 515aa88bee2e4b84b418f5704e2da830. Latest consensus state: current_term: 1 leader_uuid: "515aa88bee2e4b84b418f5704e2da830" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515aa88bee2e4b84b418f5704e2da830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } } }
I20260926 14:54:15.125583 1418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:15.126601 1419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "515aa88bee2e4b84b418f5704e2da830" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515aa88bee2e4b84b418f5704e2da830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } } }
I20260926 14:54:15.126724 1419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:15.129165 1416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:15.116014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:44005 (local address 127.0.0.1:44135)
0926 14:54:15.116439 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:15.116458 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:15.116499 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:54:15.117180 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:15.117197 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:15.117522 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:15.117826 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:15.118695 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:15.119624 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:15.120520 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:15.120972 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:15.121111 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:15.123926 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:15.123933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:15.123950 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:15.123979 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:15.126028 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:15.126587 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:15.126593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:15.126598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:15.126749 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:15.127169 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:15.127172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:15.127174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:15.127661 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:15.127867 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:15.128196 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 14:54:15.128405 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":120,"threads_started":1}
W20260926 14:54:15.129941 1434 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:54:15.130062 1434 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:54:15.130162 1423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:15.130986 1423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:15.131830 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:15.137095 1423 catalog_manager.cc:1347] Generated new cluster ID: 47a26aedc6014a28b084a27352742235
I20260926 14:54:15.137166 1423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:15.141625 1416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:15.133788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:36783 (local address 127.0.0.1:44135)
0926 14:54:15.133885 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:15.133889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:15.134132 (+ 243us) server_negotiation.cc:407] Connection header received
0926 14:54:15.134225 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:15.134228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:15.134275 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:15.134356 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:15.135073 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:15.135853 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:15.137013 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:15.137285 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:15.137378 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:15.138663 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:15.138666 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:15.138669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:15.138692 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:15.140052 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:15.140484 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:15.140488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:15.140489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:15.140564 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:15.140852 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:15.140855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:15.140857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:15.141015 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:15.141139 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:15.141404 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 14:54:15.141484 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:54:15.166498 1423 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:15.167469 1423 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:15.171778 1423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 515aa88bee2e4b84b418f5704e2da830: Generated new TSK 0
I20260926 14:54:15.172457 1423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:15.175908 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 1345
W20260926 14:54:15.181231 1330 connection.cc:535] client connection to 127.0.0.1:44135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44135): Connection reset by peer (error 104)
2026-09-26T14:54:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:15 dist-test-slave-24mn krb5kdc[1444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:15 dist-test-slave-24mn krb5kdc[1444](info): set up 2 sockets
Sep 26 14:54:15 dist-test-slave-24mn krb5kdc[1444](info): commencing operation
krb5kdc: starting...
W20260926 14:54:16.119814 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:16 dist-test-slave-24mn krb5kdc[1444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:54:16.230582 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:16.333824 1457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:16.334057 1457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:16.334108 1457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:16.338030 1457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:16.338083 1457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:16.338109 1457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:16.338128 1457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:16.338148 1457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:16.343204 1457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:16.344144 1457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:16.345091 1457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:16.352837 1462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:16.352849 1463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:16.352873 1465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:16.353082 1457 server_base.cc:1034] running on GCE node
I20260926 14:54:16.353605 1457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:16.354641 1457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:16.355803 1457 hybrid_clock.cc:648] HybridClock initialized: now 1790434456355783 us; error 30 us; skew 500 ppm
Sep 26 14:54:16 dist-test-slave-24mn krb5kdc[1444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:16.358407 1457 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:54:16.359208 1457 webserver.cc:460] Webserver started at http://127.0.0.1:45975/ using document root <none> and password file <none>
I20260926 14:54:16.359706 1457 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:16.359768 1457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:16.359979 1457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:16.361943 1457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "ec9388a09ebc4bad85ce789d9c1b81d0"
format_stamp: "Formatted at 2026-09-26 14:54:16 on dist-test-slave-24mn"
server_key: "151c3d1e8c7a53c65fbac97babd01559"
server_key_iv: "93ee80bbfd47e8b9b71c1210a72d3fdd"
server_key_version: "encryptionkey@0"
I20260926 14:54:16.362460 1457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "ec9388a09ebc4bad85ce789d9c1b81d0"
format_stamp: "Formatted at 2026-09-26 14:54:16 on dist-test-slave-24mn"
server_key: "151c3d1e8c7a53c65fbac97babd01559"
server_key_iv: "93ee80bbfd47e8b9b71c1210a72d3fdd"
server_key_version: "encryptionkey@0"
I20260926 14:54:16.366017 1457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:54:16.368216 1472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:16.369429 1457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:54:16.369544 1457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "ec9388a09ebc4bad85ce789d9c1b81d0"
format_stamp: "Formatted at 2026-09-26 14:54:16 on dist-test-slave-24mn"
server_key: "151c3d1e8c7a53c65fbac97babd01559"
server_key_iv: "93ee80bbfd47e8b9b71c1210a72d3fdd"
server_key_version: "encryptionkey@0"
I20260926 14:54:16.369635 1457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:16.394398 1457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:16.397547 1457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:16.397728 1457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:16.404588 1457 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36821
I20260926 14:54:16.404624 1524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36821 every 8 connection(s)
I20260926 14:54:16.405515 1457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:16.408181 1525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:16.413609 1525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0: Bootstrap starting.
I20260926 14:54:16.415771 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 1457
I20260926 14:54:16.415915 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:16.416122 1525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:16.416196 12563 external_mini_cluster.cc:1428] Setting key 3f361734a65079ec7590e35181fa3f73
I20260926 14:54:16.416958 1525 log.cc:824] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:16.418771 1525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0: No bootstrap required, opened a new log
I20260926 14:54:16.422302 1525 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9388a09ebc4bad85ce789d9c1b81d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36821 } }
I20260926 14:54:16.422631 1525 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:16.422722 1525 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec9388a09ebc4bad85ce789d9c1b81d0, State: Initialized, Role: FOLLOWER
I20260926 14:54:16.423205 1525 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9388a09ebc4bad85ce789d9c1b81d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36821 } }
I20260926 14:54:16.423365 1525 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:16.423434 1525 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:16.423532 1525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:16.424480 1525 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9388a09ebc4bad85ce789d9c1b81d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36821 } }
I20260926 14:54:16.424721 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:16.417241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:46099 (local address 127.0.0.1:36821)
0926 14:54:16.417649 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:16.417664 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:16.417698 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:54:16.418321 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:16.418338 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:16.418657 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:16.418954 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:16.419346 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:16.420357 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:16.421247 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:16.421700 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:16.421839 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:16.422804 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:16.422811 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:16.422827 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:16.423320 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:16.423392 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:16.423423 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:16.423560 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:16.423856 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 14:54:16.424013 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":128,"threads_started":1}
I20260926 14:54:16.424803 1525 leader_election.cc:302] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec9388a09ebc4bad85ce789d9c1b81d0; no voters:
I20260926 14:54:16.425128 1525 leader_election.cc:288] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:16.425241 1530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:16.425448 1530 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [term 1 LEADER]: Becoming Leader. State: Replica: ec9388a09ebc4bad85ce789d9c1b81d0, State: Running, Role: LEADER
I20260926 14:54:16.425789 1530 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9388a09ebc4bad85ce789d9c1b81d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36821 } }
I20260926 14:54:16.426298 1525 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:16.427438 1531 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec9388a09ebc4bad85ce789d9c1b81d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9388a09ebc4bad85ce789d9c1b81d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36821 } } }
I20260926 14:54:16.427438 1532 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec9388a09ebc4bad85ce789d9c1b81d0. Latest consensus state: current_term: 1 leader_uuid: "ec9388a09ebc4bad85ce789d9c1b81d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9388a09ebc4bad85ce789d9c1b81d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36821 } } }
I20260926 14:54:16.427608 1532 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:16.427608 1531 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:16.427722 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:16.428072 1539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:16.431224 1539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:16.436236 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:16.429850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:42181 (local address 127.0.0.1:36821)
0926 14:54:16.429942 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:16.429946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:16.432135 (+ 2189us) server_negotiation.cc:407] Connection header received
0926 14:54:16.432232 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:16.432235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:16.432279 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:16.432343 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:16.432703 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:16.433398 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:16.434479 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:16.434811 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:16.434893 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:16.435812 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:16.435815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:16.435818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:16.435829 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:16.435853 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:16.435863 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:16.435979 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:16.436080 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 14:54:16.436110 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:54:16.436985 1539 catalog_manager.cc:1347] Generated new cluster ID: 8876d9faa7f14f63b8948fcc463aae69
I20260926 14:54:16.437040 1539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:16.448434 1539 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:16.449275 1539 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:16.453886 1539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec9388a09ebc4bad85ce789d9c1b81d0: Generated new TSK 0
I20260926 14:54:16.454473 1539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:16.459408 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 1457
W20260926 14:54:16.463817 1442 connection.cc:535] client connection to 127.0.0.1:36821 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36821): Connection reset by peer (error 104)
2026-09-26T14:54:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:16 dist-test-slave-24mn krb5kdc[1556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:16 dist-test-slave-24mn krb5kdc[1556](info): set up 2 sockets
Sep 26 14:54:16 dist-test-slave-24mn krb5kdc[1556](info): commencing operation
krb5kdc: starting...
W20260926 14:54:17.411816 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:17 dist-test-slave-24mn krb5kdc[1556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:54:17.523800 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:17.626703 1569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:17.626942 1569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:17.626996 1569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:17.630900 1569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:17.630954 1569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:17.630978 1569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:17.630997 1569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:17.631018 1569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:17.636050 1569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:17.636968 1569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:17.637887 1569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:17.645809 1575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:17.645840 1574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:17.645889 1569 server_base.cc:1034] running on GCE node
W20260926 14:54:17.645809 1577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:17.646536 1569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:17.647554 1569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:17.648710 1569 hybrid_clock.cc:648] HybridClock initialized: now 1790434457648698 us; error 28 us; skew 500 ppm
Sep 26 14:54:17 dist-test-slave-24mn krb5kdc[1556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:17.651288 1569 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:54:17.652155 1569 webserver.cc:460] Webserver started at http://127.0.0.1:46037/ using document root <none> and password file <none>
I20260926 14:54:17.652659 1569 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:17.652732 1569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:17.652921 1569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:17.654835 1569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "2cc006ee015e4b92bad4d0141ad974e0"
format_stamp: "Formatted at 2026-09-26 14:54:17 on dist-test-slave-24mn"
server_key: "97ba3bea7b1d267d6376e6bd6f4a1a18"
server_key_iv: "3feac41c97c8b77552f52742b42478c3"
server_key_version: "encryptionkey@0"
I20260926 14:54:17.655339 1569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "2cc006ee015e4b92bad4d0141ad974e0"
format_stamp: "Formatted at 2026-09-26 14:54:17 on dist-test-slave-24mn"
server_key: "97ba3bea7b1d267d6376e6bd6f4a1a18"
server_key_iv: "3feac41c97c8b77552f52742b42478c3"
server_key_version: "encryptionkey@0"
I20260926 14:54:17.658800 1569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:54:17.661214 1584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:17.662503 1569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:54:17.662623 1569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "2cc006ee015e4b92bad4d0141ad974e0"
format_stamp: "Formatted at 2026-09-26 14:54:17 on dist-test-slave-24mn"
server_key: "97ba3bea7b1d267d6376e6bd6f4a1a18"
server_key_iv: "3feac41c97c8b77552f52742b42478c3"
server_key_version: "encryptionkey@0"
I20260926 14:54:17.662712 1569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:17.666886 1569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:17.669469 1569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:17.669615 1569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:17.675679 1569 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35271
I20260926 14:54:17.675724 1636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35271 every 8 connection(s)
I20260926 14:54:17.676651 1569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:17.678625 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 1569
I20260926 14:54:17.678766 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:17.679018 12563 external_mini_cluster.cc:1428] Setting key bd9011c051370c57495ccc9745603032
I20260926 14:54:17.679265 1637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:17.684135 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:17.680306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:53747 (local address 127.0.0.1:35271)
0926 14:54:17.680693 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:17.680711 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:17.680750 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:54:17.681444 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:17.681464 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:17.681800 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:17.682157 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:17.682823 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:17.682829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:17.682844 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:17.682864 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:17.682922 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:17.682947 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:17.683080 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:17.683144 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 14:54:17.683282 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20260926 14:54:17.685971 1637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0: Bootstrap starting.
I20260926 14:54:17.686794 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:17.688176 1637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:17.688879 1637 log.cc:824] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:17.690152 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:17.688645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:37731 (local address 127.0.0.1:35271)
0926 14:54:17.688758 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:17.688762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:17.688969 (+ 207us) server_negotiation.cc:407] Connection header received
0926 14:54:17.689070 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:17.689073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:17.689110 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:17.689192 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:17.689831 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:17.689834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:17.689837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:17.689848 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:17.689873 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:17.689885 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:17.689961 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:17.690016 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 14:54:17.690049 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:54:17.690929 1637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0: No bootstrap required, opened a new log
I20260926 14:54:17.695497 1637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc006ee015e4b92bad4d0141ad974e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35271 } }
I20260926 14:54:17.695770 1637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:17.695832 1637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc006ee015e4b92bad4d0141ad974e0, State: Initialized, Role: FOLLOWER
I20260926 14:54:17.696312 1637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc006ee015e4b92bad4d0141ad974e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35271 } }
I20260926 14:54:17.696475 1637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:17.696544 1637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:17.696642 1637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:17.697669 1637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc006ee015e4b92bad4d0141ad974e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35271 } }
I20260926 14:54:17.697947 1637 leader_election.cc:302] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cc006ee015e4b92bad4d0141ad974e0; no voters:
I20260926 14:54:17.698170 1637 leader_election.cc:288] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:17.698259 1647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:17.698451 1647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [term 1 LEADER]: Becoming Leader. State: Replica: 2cc006ee015e4b92bad4d0141ad974e0, State: Running, Role: LEADER
I20260926 14:54:17.698764 1647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc006ee015e4b92bad4d0141ad974e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35271 } }
I20260926 14:54:17.699265 1637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:17.700265 1649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cc006ee015e4b92bad4d0141ad974e0. Latest consensus state: current_term: 1 leader_uuid: "2cc006ee015e4b92bad4d0141ad974e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc006ee015e4b92bad4d0141ad974e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35271 } } }
I20260926 14:54:17.700388 1649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:17.700728 1656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:17.701458 1648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cc006ee015e4b92bad4d0141ad974e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc006ee015e4b92bad4d0141ad974e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35271 } } }
I20260926 14:54:17.701534 1648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:17.703284 1656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:17.708549 1656 catalog_manager.cc:1347] Generated new cluster ID: bb6a94dd584f489ab0a936df254f02c5
I20260926 14:54:17.708623 1656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:17.725454 1656 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:17.725708 1656 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:17.730014 1656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cc006ee015e4b92bad4d0141ad974e0: Generated new TSK 0
I20260926 14:54:17.730551 1656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:17.738101 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 1569
2026-09-26T14:54:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:17 dist-test-slave-24mn krb5kdc[1668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:17 dist-test-slave-24mn krb5kdc[1668](info): set up 2 sockets
Sep 26 14:54:17 dist-test-slave-24mn krb5kdc[1668](info): commencing operation
krb5kdc: starting...
W20260926 14:54:18.687798 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:18 dist-test-slave-24mn krb5kdc[1668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:54:18.799115 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:35019
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:38975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:35019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:18.903580 1681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:18.903846 1681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:18.903927 1681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:18.908012 1681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:18.908074 1681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:18.908092 1681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:18.908107 1681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:18.908120 1681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:18.913331 1681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:38975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:35019
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:35019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:18.914248 1681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:18.915146 1681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:18.923022 1687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:18.923033 1686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:18.923085 1681 server_base.cc:1034] running on GCE node
W20260926 14:54:18.923039 1689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:18.923813 1681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:18.924875 1681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:18.926029 1681 hybrid_clock.cc:648] HybridClock initialized: now 1790434458926015 us; error 28 us; skew 500 ppm
Sep 26 14:54:18 dist-test-slave-24mn krb5kdc[1668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434458, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:18.931764 1681 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:54:18.932631 1681 webserver.cc:460] Webserver started at http://127.12.68.254:45055/ using document root <none> and password file <none>
I20260926 14:54:18.933156 1681 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:18.933213 1681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:18.933403 1681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:18.935377 1681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "6afba9c7537144fdbe1cd71da889cfae"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-24mn"
server_key: "9a1149e616c6f5e9ea956cb7eb4a1b9d"
server_key_iv: "a3c1074cb5c16123471f4b9cfb87abc5"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.935930 1681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "6afba9c7537144fdbe1cd71da889cfae"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-24mn"
server_key: "9a1149e616c6f5e9ea956cb7eb4a1b9d"
server_key_iv: "a3c1074cb5c16123471f4b9cfb87abc5"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.939412 1681 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 14:54:18.941824 1696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:18.942999 1681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:18.943107 1681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "6afba9c7537144fdbe1cd71da889cfae"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-24mn"
server_key: "9a1149e616c6f5e9ea956cb7eb4a1b9d"
server_key_iv: "a3c1074cb5c16123471f4b9cfb87abc5"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.943207 1681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:18.973882 1681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:18.976969 1681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:18.977142 1681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:18.983820 1681 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:35019
I20260926 14:54:18.983850 1748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:35019 every 8 connection(s)
I20260926 14:54:18.984757 1681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:18.987406 1749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:18.992887 1749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae: Bootstrap starting.
I20260926 14:54:18.994549 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 1681
I20260926 14:54:18.994661 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:18.994925 12563 external_mini_cluster.cc:1428] Setting key b03b63cc3cecdfc3c0bf469dc16031b7
I20260926 14:54:18.995432 1749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:18.996277 1749 log.cc:824] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:18.998456 1749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae: No bootstrap required, opened a new log
I20260926 14:54:19.002691 1749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afba9c7537144fdbe1cd71da889cfae" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35019 } }
I20260926 14:54:19.002912 1749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:19.002956 1749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6afba9c7537144fdbe1cd71da889cfae, State: Initialized, Role: FOLLOWER
I20260926 14:54:19.003460 1749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afba9c7537144fdbe1cd71da889cfae" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35019 } }
I20260926 14:54:19.003588 1749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:19.003648 1749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:19.003744 1749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [term 0 FOLLOWER]: Advancing to term 1
Sep 26 14:54:19 dist-test-slave-24mn krb5kdc[1668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:54:19.004688 1749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afba9c7537144fdbe1cd71da889cfae" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35019 } }
I20260926 14:54:19.005059 1749 leader_election.cc:302] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6afba9c7537144fdbe1cd71da889cfae; no voters:
I20260926 14:54:19.005407 1749 leader_election.cc:288] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:19.005491 1754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:19.005702 1754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [term 1 LEADER]: Becoming Leader. State: Replica: 6afba9c7537144fdbe1cd71da889cfae, State: Running, Role: LEADER
I20260926 14:54:19.006071 1754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afba9c7537144fdbe1cd71da889cfae" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35019 } }
I20260926 14:54:19.006738 1749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:19.007504 1755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6afba9c7537144fdbe1cd71da889cfae. Latest consensus state: current_term: 1 leader_uuid: "6afba9c7537144fdbe1cd71da889cfae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afba9c7537144fdbe1cd71da889cfae" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35019 } } }
I20260926 14:54:19.007623 1755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:19.008111 1763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:19.008983 1756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6afba9c7537144fdbe1cd71da889cfae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afba9c7537144fdbe1cd71da889cfae" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35019 } } }
I20260926 14:54:19.009120 1756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:19.010692 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.996064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43499 (local address 127.12.68.254:35019)
0926 14:54:18.996553 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:18.996578 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:18.996622 (+ 44us) server_negotiation.cc:407] Connection header received
0926 14:54:18.997399 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:18.997421 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:18.997845 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:18.998217 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:18.998993 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:19.000516 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:19.001433 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:19.002144 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:19.002312 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:19.004784 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:19.004791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:19.004808 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:19.004839 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:19.006778 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:19.007387 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:19.007394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:19.007399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:19.007548 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:19.007998 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:19.008002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:19.008003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:19.008525 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:19.008764 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:19.009535 (+ 771us) server_negotiation.cc:299] Negotiation successful
0926 14:54:19.009770 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":150,"threads_started":1}
I20260926 14:54:19.011312 1763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:19.013839 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:19.017673 1763 catalog_manager.cc:1347] Generated new cluster ID: 6e89a8b0a6c7460e860a4d36f99ca32e
I20260926 14:54:19.017729 1763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:19.024365 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:19.016073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33145 (local address 127.12.68.254:35019)
0926 14:54:19.016173 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:19.016177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:19.016343 (+ 166us) server_negotiation.cc:407] Connection header received
0926 14:54:19.016460 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:19.016463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:19.016504 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:19.016566 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:19.017189 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:19.017893 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:19.019058 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:19.019412 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:19.019502 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:19.021084 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:19.021088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:19.021090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:19.021113 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:19.022627 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:19.023124 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:19.023128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:19.023129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:19.023206 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:19.023511 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:19.023514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:19.023516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:19.023703 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:19.023843 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:19.024126 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 14:54:19.024217 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":37}
I20260926 14:54:19.026253 1763 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:19.027072 1763 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:19.032395 1763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6afba9c7537144fdbe1cd71da889cfae: Generated new TSK 0
I20260926 14:54:19.033087 1763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:19.040922 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 1681
W20260926 14:54:19.045534 1666 connection.cc:535] client connection to 127.12.68.254:35019 recv error: Network error: failed to read from TLS socket (remote: 127.12.68.254:35019): Connection reset by peer (error 104)
2026-09-26T14:54:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:19 dist-test-slave-24mn krb5kdc[1780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:19 dist-test-slave-24mn krb5kdc[1780](info): set up 2 sockets
Sep 26 14:54:19 dist-test-slave-24mn krb5kdc[1780](info): commencing operation
krb5kdc: starting...
W20260926 14:54:20.003755 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:20 dist-test-slave-24mn krb5kdc[1780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:54:20.114974 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:35803
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:35803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:20.222083 1793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:20.222326 1793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:20.222366 1793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:20.226367 1793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:20.226425 1793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:20.226444 1793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:20.226459 1793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:20.226472 1793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:20.231655 1793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:35803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:35803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:20.232647 1793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:20.233621 1793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:20.241457 1799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:20.241511 1793 server_base.cc:1034] running on GCE node
W20260926 14:54:20.241478 1798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:20.241479 1801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:20.242223 1793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:20.243253 1793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:20.244419 1793 hybrid_clock.cc:648] HybridClock initialized: now 1790434460244404 us; error 28 us; skew 500 ppm
Sep 26 14:54:20 dist-test-slave-24mn krb5kdc[1780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434460, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:20.246938 1793 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:54:20.247788 1793 webserver.cc:460] Webserver started at http://127.12.68.254:41997/ using document root <none> and password file <none>
I20260926 14:54:20.248356 1793 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:20.248425 1793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:20.248602 1793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:20.250588 1793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "c471e1ba2d8c4267bdbd54e7aea4296d"
format_stamp: "Formatted at 2026-09-26 14:54:20 on dist-test-slave-24mn"
server_key: "e5e70d3aa42bef8e21bc931db04fbb24"
server_key_iv: "1d43cae5116316c2e4df063e8310c5bf"
server_key_version: "encryptionkey@0"
I20260926 14:54:20.251104 1793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "c471e1ba2d8c4267bdbd54e7aea4296d"
format_stamp: "Formatted at 2026-09-26 14:54:20 on dist-test-slave-24mn"
server_key: "e5e70d3aa42bef8e21bc931db04fbb24"
server_key_iv: "1d43cae5116316c2e4df063e8310c5bf"
server_key_version: "encryptionkey@0"
I20260926 14:54:20.254650 1793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:54:20.256886 1808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:20.258024 1793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:54:20.258132 1793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "c471e1ba2d8c4267bdbd54e7aea4296d"
format_stamp: "Formatted at 2026-09-26 14:54:20 on dist-test-slave-24mn"
server_key: "e5e70d3aa42bef8e21bc931db04fbb24"
server_key_iv: "1d43cae5116316c2e4df063e8310c5bf"
server_key_version: "encryptionkey@0"
I20260926 14:54:20.258215 1793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:20.275571 1793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:20.278280 1793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:20.278446 1793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:20.285001 1793 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:35803
I20260926 14:54:20.285017 1860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:35803 every 8 connection(s)
I20260926 14:54:20.285945 1793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:20.288584 1861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:20.290484 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 1793
I20260926 14:54:20.290614 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:20.290864 12563 external_mini_cluster.cc:1428] Setting key cfcd27108e01c5a40b96b9379a65910e
I20260926 14:54:20.294728 1861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d: Bootstrap starting.
I20260926 14:54:20.297258 1861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:20.298120 1861 log.cc:824] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:20.300493 1861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d: No bootstrap required, opened a new log
I20260926 14:54:20.300657 1864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:20.292245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60011 (local address 127.12.68.254:35803)
0926 14:54:20.292649 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:20.292668 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:20.292710 (+ 42us) server_negotiation.cc:407] Connection header received
0926 14:54:20.293651 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:20.293675 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:20.294094 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:20.294460 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:20.294849 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:20.296282 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:20.297179 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:20.297858 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:20.298019 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:20.298751 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:20.298757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:20.298773 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:20.299196 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:20.299257 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:20.299286 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:20.299406 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:20.299593 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 14:54:20.299721 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":99,"threads_started":1}
I20260926 14:54:20.303381 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:20.304176 1861 raft_consensus.cc:348] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c471e1ba2d8c4267bdbd54e7aea4296d" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35803 } }
I20260926 14:54:20.304450 1861 raft_consensus.cc:374] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:20.304556 1861 raft_consensus.cc:729] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c471e1ba2d8c4267bdbd54e7aea4296d, State: Initialized, Role: FOLLOWER
I20260926 14:54:20.305112 1861 consensus_queue.cc:260] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c471e1ba2d8c4267bdbd54e7aea4296d" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35803 } }
I20260926 14:54:20.305323 1861 raft_consensus.cc:388] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:20.305423 1861 raft_consensus.cc:482] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:20.305531 1861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:20.306591 1861 raft_consensus.cc:504] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c471e1ba2d8c4267bdbd54e7aea4296d" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35803 } }
I20260926 14:54:20.306948 1861 leader_election.cc:302] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c471e1ba2d8c4267bdbd54e7aea4296d; no voters:
I20260926 14:54:20.307232 1861 leader_election.cc:288] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:20.307318 1871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:20.307484 1871 raft_consensus.cc:686] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [term 1 LEADER]: Becoming Leader. State: Replica: c471e1ba2d8c4267bdbd54e7aea4296d, State: Running, Role: LEADER
I20260926 14:54:20.307817 1871 consensus_queue.cc:237] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c471e1ba2d8c4267bdbd54e7aea4296d" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35803 } }
I20260926 14:54:20.308491 1861 sys_catalog.cc:564] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:20.309031 1871 sys_catalog.cc:455] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c471e1ba2d8c4267bdbd54e7aea4296d. Latest consensus state: current_term: 1 leader_uuid: "c471e1ba2d8c4267bdbd54e7aea4296d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c471e1ba2d8c4267bdbd54e7aea4296d" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35803 } } }
I20260926 14:54:20.309203 1871 sys_catalog.cc:458] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:20.309420 1872 sys_catalog.cc:455] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c471e1ba2d8c4267bdbd54e7aea4296d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c471e1ba2d8c4267bdbd54e7aea4296d" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35803 } } }
I20260926 14:54:20.309521 1872 sys_catalog.cc:458] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:20.309796 1879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:20.309973 1864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:20.305337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36571 (local address 127.12.68.254:35803)
0926 14:54:20.305476 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:20.305480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:20.305655 (+ 175us) server_negotiation.cc:407] Connection header received
0926 14:54:20.305727 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:20.305730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:20.305773 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:20.305851 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:20.306068 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:20.306812 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:20.307744 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:20.308051 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:20.308141 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:20.308906 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:20.308910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:20.308913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:20.308925 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:20.308952 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:20.308964 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:20.309053 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:20.309806 (+ 753us) server_negotiation.cc:299] Negotiation successful
0926 14:54:20.309842 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:54:20.312798 1879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:20.317553 1879 catalog_manager.cc:1347] Generated new cluster ID: 875308ed3d6e4a9f977d7e1be7cef5a5
I20260926 14:54:20.317618 1879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:20.334983 1879 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:20.336205 1879 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:20.346956 1879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c471e1ba2d8c4267bdbd54e7aea4296d: Generated new TSK 0
I20260926 14:54:20.347647 1879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:20.356009 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 1793
W20260926 14:54:20.360822 1778 connection.cc:535] client connection to 127.12.68.254:35803 recv error: Network error: failed to read from TLS socket (remote: 127.12.68.254:35803): Connection reset by peer (error 104)
2026-09-26T14:54:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:20 dist-test-slave-24mn krb5kdc[1892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:20 dist-test-slave-24mn krb5kdc[1892](info): set up 2 sockets
Sep 26 14:54:20 dist-test-slave-24mn krb5kdc[1892](info): commencing operation
krb5kdc: starting...
W20260926 14:54:21.303761 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:21 dist-test-slave-24mn krb5kdc[1892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:54:21.415114 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:39639
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:39639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:21.518302 1905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:21.518541 1905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:21.518581 1905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:21.522586 1905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:21.522645 1905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:21.522663 1905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:21.522678 1905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:21.522692 1905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:21.527815 1905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:39639
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:39639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:21.528805 1905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:21.529780 1905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:21.537746 1910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:21.537756 1913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:21.537755 1911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:21.537803 1905 server_base.cc:1034] running on GCE node
I20260926 14:54:21.538480 1905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:21.539515 1905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:21.540658 1905 hybrid_clock.cc:648] HybridClock initialized: now 1790434461540632 us; error 38 us; skew 500 ppm
Sep 26 14:54:21 dist-test-slave-24mn krb5kdc[1892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434461, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:21.543069 1905 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:54:21.543906 1905 webserver.cc:460] Webserver started at http://127.12.68.254:40453/ using document root <none> and password file <none>
I20260926 14:54:21.544412 1905 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:21.544476 1905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:21.544654 1905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:21.546499 1905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "3e33ab684f7c4dfab7b0b9cfe0b1d221"
format_stamp: "Formatted at 2026-09-26 14:54:21 on dist-test-slave-24mn"
server_key: "a604fe4ebbf73c3c6b8b84c138912ff6"
server_key_iv: "310677fbefb88862a3f8408c32dcc51f"
server_key_version: "encryptionkey@0"
I20260926 14:54:21.546988 1905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "3e33ab684f7c4dfab7b0b9cfe0b1d221"
format_stamp: "Formatted at 2026-09-26 14:54:21 on dist-test-slave-24mn"
server_key: "a604fe4ebbf73c3c6b8b84c138912ff6"
server_key_iv: "310677fbefb88862a3f8408c32dcc51f"
server_key_version: "encryptionkey@0"
I20260926 14:54:21.550767 1905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:54:21.553005 1920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:21.554131 1905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:54:21.554248 1905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "3e33ab684f7c4dfab7b0b9cfe0b1d221"
format_stamp: "Formatted at 2026-09-26 14:54:21 on dist-test-slave-24mn"
server_key: "a604fe4ebbf73c3c6b8b84c138912ff6"
server_key_iv: "310677fbefb88862a3f8408c32dcc51f"
server_key_version: "encryptionkey@0"
I20260926 14:54:21.554337 1905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:21.558646 1905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:21.561362 1905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:21.561522 1905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:21.567816 1905 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:39639
I20260926 14:54:21.567834 1972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:39639 every 8 connection(s)
I20260926 14:54:21.568673 1905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:21.570014 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 1905
I20260926 14:54:21.570139 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:21.570387 12563 external_mini_cluster.cc:1428] Setting key 8c2ed46491dd161641a1aeeb12bb05dc
I20260926 14:54:21.571431 1973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:21.575415 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:21.571649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32873 (local address 127.12.68.254:39639)
0926 14:54:21.572262 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:21.572283 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:21.572325 (+ 42us) server_negotiation.cc:407] Connection header received
0926 14:54:21.572984 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:21.573000 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:21.573273 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:21.573540 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:21.574307 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:21.574312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:21.574324 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:21.574334 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:21.574382 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:21.574399 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:21.574507 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:21.574612 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 14:54:21.574744 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":150,"threads_started":1}
I20260926 14:54:21.577350 1973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221: Bootstrap starting.
I20260926 14:54:21.578078 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:21.579820 1973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:21.580665 1973 log.cc:824] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:21.581532 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:21.580126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40631 (local address 127.12.68.254:39639)
0926 14:54:21.580238 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:21.580242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:21.580375 (+ 133us) server_negotiation.cc:407] Connection header received
0926 14:54:21.580477 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:21.580480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:21.580511 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:21.580578 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:21.581188 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:21.581191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:21.581193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:21.581204 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:21.581228 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:21.581240 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:21.581320 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:21.581415 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 14:54:21.581442 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:54:21.582542 1973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221: No bootstrap required, opened a new log
I20260926 14:54:21.585755 1973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e33ab684f7c4dfab7b0b9cfe0b1d221" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 39639 } }
I20260926 14:54:21.585956 1973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:21.586019 1973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e33ab684f7c4dfab7b0b9cfe0b1d221, State: Initialized, Role: FOLLOWER
I20260926 14:54:21.586386 1973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e33ab684f7c4dfab7b0b9cfe0b1d221" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 39639 } }
I20260926 14:54:21.586520 1973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:21.586581 1973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:21.586674 1973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:21.587451 1973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e33ab684f7c4dfab7b0b9cfe0b1d221" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 39639 } }
I20260926 14:54:21.587743 1973 leader_election.cc:302] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e33ab684f7c4dfab7b0b9cfe0b1d221; no voters:
I20260926 14:54:21.588048 1973 leader_election.cc:288] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:21.588145 1983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:21.588354 1983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [term 1 LEADER]: Becoming Leader. State: Replica: 3e33ab684f7c4dfab7b0b9cfe0b1d221, State: Running, Role: LEADER
I20260926 14:54:21.588624 1983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e33ab684f7c4dfab7b0b9cfe0b1d221" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 39639 } }
I20260926 14:54:21.589116 1973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:21.590059 1984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e33ab684f7c4dfab7b0b9cfe0b1d221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e33ab684f7c4dfab7b0b9cfe0b1d221" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 39639 } } }
I20260926 14:54:21.590180 1984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:21.590363 1985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e33ab684f7c4dfab7b0b9cfe0b1d221. Latest consensus state: current_term: 1 leader_uuid: "3e33ab684f7c4dfab7b0b9cfe0b1d221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e33ab684f7c4dfab7b0b9cfe0b1d221" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 39639 } } }
I20260926 14:54:21.590457 1985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:21.590695 1992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:21.593567 1992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:21.598352 1992 catalog_manager.cc:1347] Generated new cluster ID: 68815480d5554bd1a6449e9ff5a88ba6
I20260926 14:54:21.598412 1992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:21.610882 1992 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:21.611187 1992 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:21.616283 1992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e33ab684f7c4dfab7b0b9cfe0b1d221: Generated new TSK 0
I20260926 14:54:21.616956 1992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:21.627553 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 1905
2026-09-26T14:54:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:21 dist-test-slave-24mn krb5kdc[2004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:21 dist-test-slave-24mn krb5kdc[2004](info): set up 2 sockets
Sep 26 14:54:21 dist-test-slave-24mn krb5kdc[2004](info): commencing operation
krb5kdc: starting...
W20260926 14:54:22.575805 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:22 dist-test-slave-24mn krb5kdc[2004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:54:22.687083 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:46097
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:46097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:22.790442 2017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:22.790668 2017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:22.790704 2017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:22.795866 2017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:22.795949 2017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:22.795977 2017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:22.796007 2017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:22.796027 2017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:22.802042 2017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:46097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:46097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:22.802987 2017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:22.803995 2017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:22.811836 2023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:22.811982 2017 server_base.cc:1034] running on GCE node
W20260926 14:54:22.811911 2022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:22.811841 2025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:22.812623 2017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:22.813576 2017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:22.814726 2017 hybrid_clock.cc:648] HybridClock initialized: now 1790434462814706 us; error 31 us; skew 500 ppm
Sep 26 14:54:22 dist-test-slave-24mn krb5kdc[2004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434462, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:22.817227 2017 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:54:22.818002 2017 webserver.cc:460] Webserver started at http://127.12.68.254:45113/ using document root <none> and password file <none>
I20260926 14:54:22.818487 2017 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:22.818548 2017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:22.818724 2017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:22.820765 2017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "ca9be40fc87e4bbf90b0d4f0a5264a87"
format_stamp: "Formatted at 2026-09-26 14:54:22 on dist-test-slave-24mn"
server_key: "1bfe89d8f6f4f99e84bd7b11cb791435"
server_key_iv: "e31524293eaed009179cd72b70aafdf4"
server_key_version: "encryptionkey@0"
I20260926 14:54:22.821290 2017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "ca9be40fc87e4bbf90b0d4f0a5264a87"
format_stamp: "Formatted at 2026-09-26 14:54:22 on dist-test-slave-24mn"
server_key: "1bfe89d8f6f4f99e84bd7b11cb791435"
server_key_iv: "e31524293eaed009179cd72b70aafdf4"
server_key_version: "encryptionkey@0"
I20260926 14:54:22.824654 2017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:54:22.826797 2032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:22.827863 2017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:22.828003 2017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "ca9be40fc87e4bbf90b0d4f0a5264a87"
format_stamp: "Formatted at 2026-09-26 14:54:22 on dist-test-slave-24mn"
server_key: "1bfe89d8f6f4f99e84bd7b11cb791435"
server_key_iv: "e31524293eaed009179cd72b70aafdf4"
server_key_version: "encryptionkey@0"
I20260926 14:54:22.828087 2017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:22.836607 2017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:22.839321 2017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:22.839478 2017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:22.845623 2017 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:46097
I20260926 14:54:22.845640 2084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:46097 every 8 connection(s)
I20260926 14:54:22.846462 2017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:22.849035 2085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:22.852064 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 2017
I20260926 14:54:22.852200 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:22.852470 12563 external_mini_cluster.cc:1428] Setting key 31d4a3f2dcded3b4ae97513be1533e1f
I20260926 14:54:22.854532 2085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87: Bootstrap starting.
I20260926 14:54:22.856778 2085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:22.857496 2085 log.cc:824] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:22.859383 2085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87: No bootstrap required, opened a new log
Sep 26 14:54:22 dist-test-slave-24mn krb5kdc[2004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:54:22.862522 2085 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9be40fc87e4bbf90b0d4f0a5264a87" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 46097 } }
I20260926 14:54:22.862720 2085 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:22.862774 2085 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9be40fc87e4bbf90b0d4f0a5264a87, State: Initialized, Role: FOLLOWER
I20260926 14:54:22.863173 2085 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9be40fc87e4bbf90b0d4f0a5264a87" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 46097 } }
I20260926 14:54:22.863291 2085 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:22.863329 2085 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:22.863415 2085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:22.864213 2085 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9be40fc87e4bbf90b0d4f0a5264a87" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 46097 } }
I20260926 14:54:22.864516 2085 leader_election.cc:302] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca9be40fc87e4bbf90b0d4f0a5264a87; no voters:
I20260926 14:54:22.864800 2085 leader_election.cc:288] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:22.864851 2090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:22.865046 2090 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [term 1 LEADER]: Becoming Leader. State: Replica: ca9be40fc87e4bbf90b0d4f0a5264a87, State: Running, Role: LEADER
I20260926 14:54:22.865324 2090 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9be40fc87e4bbf90b0d4f0a5264a87" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 46097 } }
I20260926 14:54:22.865999 2085 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:22.866329 2088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:22.853689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50535 (local address 127.12.68.254:46097)
0926 14:54:22.854048 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:22.854063 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:22.854095 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:54:22.854710 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:22.854728 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:22.855056 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:22.855317 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:22.856169 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:22.857174 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:22.858194 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:22.858639 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:22.858748 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:22.861410 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:22.861415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:22.861428 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:22.861462 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:22.863462 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:22.863860 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:22.863866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:22.863872 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:22.864023 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:22.864272 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:22.864276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:22.864278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:22.864687 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:22.864876 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:22.865302 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 14:54:22.865589 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":92,"threads_started":1}
I20260926 14:54:22.866667 2090 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca9be40fc87e4bbf90b0d4f0a5264a87. Latest consensus state: current_term: 1 leader_uuid: "ca9be40fc87e4bbf90b0d4f0a5264a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9be40fc87e4bbf90b0d4f0a5264a87" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 46097 } } }
I20260926 14:54:22.867534 2090 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:22.867519 2091 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca9be40fc87e4bbf90b0d4f0a5264a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9be40fc87e4bbf90b0d4f0a5264a87" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 46097 } } }
I20260926 14:54:22.867609 2091 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:22.868486 2094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:22.869436 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:22.871409 2094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:22.876816 2094 catalog_manager.cc:1347] Generated new cluster ID: fecafd8d71964b00a3d2d4976c23f85e
I20260926 14:54:22.876874 2094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:22.880882 2088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:22.871632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58335 (local address 127.12.68.254:46097)
0926 14:54:22.871728 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:22.871732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:22.873531 (+ 1799us) server_negotiation.cc:407] Connection header received
0926 14:54:22.873625 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:22.873628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:22.873670 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:22.873745 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:22.874483 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:22.875231 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:22.876401 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:22.876733 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:22.876844 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:22.878202 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:22.878206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:22.878209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:22.878232 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:22.879460 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:22.879819 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:22.879822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:22.879824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:22.879922 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:22.880152 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:22.880155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:22.880157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:22.880324 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:22.880433 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:22.880645 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 14:54:22.880748 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:54:22.890717 2094 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:22.891566 2094 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:22.896804 2094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca9be40fc87e4bbf90b0d4f0a5264a87: Generated new TSK 0
I20260926 14:54:22.897405 2094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:22.906937 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 2017
2026-09-26T14:54:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:22 dist-test-slave-24mn krb5kdc[2116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:22 dist-test-slave-24mn krb5kdc[2116](info): set up 2 sockets
Sep 26 14:54:22 dist-test-slave-24mn krb5kdc[2116](info): commencing operation
krb5kdc: starting...
W20260926 14:54:23.859838 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:23 dist-test-slave-24mn krb5kdc[2116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:54:23.974836 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:37205
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:37205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:24.079428 2129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:24.079666 2129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:24.079712 2129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:24.083631 2129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:24.083684 2129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:24.083702 2129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:24.083717 2129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:24.083735 2129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:24.088961 2129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:37205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:37205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:24.089958 2129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:24.090893 2129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:24.098708 2135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:24.098716 2137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:24.098850 2129 server_base.cc:1034] running on GCE node
W20260926 14:54:24.098749 2134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:24.099514 2129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:24.100567 2129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:24.101727 2129 hybrid_clock.cc:648] HybridClock initialized: now 1790434464101711 us; error 28 us; skew 500 ppm
Sep 26 14:54:24 dist-test-slave-24mn krb5kdc[2116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434464, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:24.108716 2129 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:54:24.109534 2129 webserver.cc:460] Webserver started at http://127.12.68.254:33855/ using document root <none> and password file <none>
I20260926 14:54:24.110036 2129 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:24.110098 2129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:24.110255 2129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:24.112241 2129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "db7fde8328c54f978a284247d6cb32fc"
format_stamp: "Formatted at 2026-09-26 14:54:24 on dist-test-slave-24mn"
server_key: "036934588ad372cd6d4fceb6752b45ba"
server_key_iv: "ca7359f8432ef133fdf6c3089341c6aa"
server_key_version: "encryptionkey@0"
I20260926 14:54:24.112730 2129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "db7fde8328c54f978a284247d6cb32fc"
format_stamp: "Formatted at 2026-09-26 14:54:24 on dist-test-slave-24mn"
server_key: "036934588ad372cd6d4fceb6752b45ba"
server_key_iv: "ca7359f8432ef133fdf6c3089341c6aa"
server_key_version: "encryptionkey@0"
I20260926 14:54:24.116302 2129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:54:24.118487 2144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:24.119611 2129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:24.119705 2129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "db7fde8328c54f978a284247d6cb32fc"
format_stamp: "Formatted at 2026-09-26 14:54:24 on dist-test-slave-24mn"
server_key: "036934588ad372cd6d4fceb6752b45ba"
server_key_iv: "ca7359f8432ef133fdf6c3089341c6aa"
server_key_version: "encryptionkey@0"
I20260926 14:54:24.119803 2129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:24.141234 2129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:24.144313 2129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:24.144483 2129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:24.150925 2129 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:37205
I20260926 14:54:24.150949 2196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:37205 every 8 connection(s)
I20260926 14:54:24.151827 2129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:24.154507 2197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:24.159669 2197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc: Bootstrap starting.
I20260926 14:54:24.160418 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 2129
I20260926 14:54:24.160568 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:24.160844 12563 external_mini_cluster.cc:1428] Setting key 29431e72a0f958e74765e49c5f016f90
I20260926 14:54:24.162454 2197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:24.163148 2197 log.cc:824] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:24.165156 2197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc: No bootstrap required, opened a new log
I20260926 14:54:24.169049 2197 raft_consensus.cc:348] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7fde8328c54f978a284247d6cb32fc" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37205 } }
I20260926 14:54:24.169268 2197 raft_consensus.cc:374] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:24.169308 2197 raft_consensus.cc:729] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7fde8328c54f978a284247d6cb32fc, State: Initialized, Role: FOLLOWER
I20260926 14:54:24.169615 2200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:24.161973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53137 (local address 127.12.68.254:37205)
0926 14:54:24.162433 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:24.162449 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:24.162485 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:54:24.163160 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:24.163177 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:24.163579 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:24.163983 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:24.164299 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:24.165259 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:24.166411 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:24.166882 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:24.166998 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:24.167852 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:24.167857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:24.167869 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:24.168274 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:24.168336 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:24.168360 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:24.168438 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:24.168575 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 14:54:24.168770 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":113,"threads_started":1}
I20260926 14:54:24.169703 2197 consensus_queue.cc:260] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7fde8328c54f978a284247d6cb32fc" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37205 } }
I20260926 14:54:24.169886 2197 raft_consensus.cc:388] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:24.169945 2197 raft_consensus.cc:482] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:24.170035 2197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:24.170936 2197 raft_consensus.cc:504] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7fde8328c54f978a284247d6cb32fc" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37205 } }
I20260926 14:54:24.171276 2197 leader_election.cc:302] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db7fde8328c54f978a284247d6cb32fc; no voters:
I20260926 14:54:24.171607 2197 leader_election.cc:288] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:24.171684 2202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:24.171912 2202 raft_consensus.cc:686] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [term 1 LEADER]: Becoming Leader. State: Replica: db7fde8328c54f978a284247d6cb32fc, State: Running, Role: LEADER
I20260926 14:54:24.172274 2202 consensus_queue.cc:237] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7fde8328c54f978a284247d6cb32fc" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37205 } }
I20260926 14:54:24.172456 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:24.172821 2197 sys_catalog.cc:564] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:24.173941 2204 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader db7fde8328c54f978a284247d6cb32fc. Latest consensus state: current_term: 1 leader_uuid: "db7fde8328c54f978a284247d6cb32fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7fde8328c54f978a284247d6cb32fc" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37205 } } }
I20260926 14:54:24.173941 2203 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db7fde8328c54f978a284247d6cb32fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7fde8328c54f978a284247d6cb32fc" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 37205 } } }
I20260926 14:54:24.174098 2203 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:24.174098 2204 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:24.174618 2215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:24.178766 2215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:24.178983 2200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:24.175283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51529 (local address 127.12.68.254:37205)
0926 14:54:24.175382 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:24.175386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:24.175555 (+ 169us) server_negotiation.cc:407] Connection header received
0926 14:54:24.175735 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:24.175738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:24.175795 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:24.175857 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:24.176144 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:24.176665 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:24.177529 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:24.177765 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:24.177849 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:24.178596 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:24.178600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:24.178602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:24.178615 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:24.178643 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:24.178653 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:24.178710 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:24.178796 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 14:54:24.178849 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 14:54:24.185006 2215 catalog_manager.cc:1347] Generated new cluster ID: 13654c1a528642a49ac57bd549678fee
I20260926 14:54:24.185086 2215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:24.202648 2215 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:24.203965 2215 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:24.210507 2215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db7fde8328c54f978a284247d6cb32fc: Generated new TSK 0
I20260926 14:54:24.211300 2215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:24.218591 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 2129
2026-09-26T14:54:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:24 dist-test-slave-24mn krb5kdc[2228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:24 dist-test-slave-24mn krb5kdc[2228](info): set up 2 sockets
Sep 26 14:54:24 dist-test-slave-24mn krb5kdc[2228](info): commencing operation
krb5kdc: starting...
W20260926 14:54:25.167833 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:25 dist-test-slave-24mn krb5kdc[2228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:54:25.279289 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:35563
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:35563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:25.382251 2241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:25.382498 2241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:25.382555 2241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:25.386476 2241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:25.386529 2241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:25.386555 2241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:25.386577 2241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:25.386605 2241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:25.391602 2241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:35563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:35563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:25.392534 2241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:25.393427 2241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:25.401264 2247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:25.401376 2241 server_base.cc:1034] running on GCE node
W20260926 14:54:25.401283 2246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:25.401283 2249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:25.402036 2241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:25.403043 2241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:25.404192 2241 hybrid_clock.cc:648] HybridClock initialized: now 1790434465404179 us; error 25 us; skew 500 ppm
Sep 26 14:54:25 dist-test-slave-24mn krb5kdc[2228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:25.406656 2241 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:54:25.407454 2241 webserver.cc:460] Webserver started at http://127.12.68.254:33245/ using document root <none> and password file <none>
I20260926 14:54:25.407989 2241 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:25.408056 2241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:25.408241 2241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:25.410135 2241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "502ab9218dce46febcb84279ae752800"
format_stamp: "Formatted at 2026-09-26 14:54:25 on dist-test-slave-24mn"
server_key: "f7a3495127096046730ef966c4bac1b1"
server_key_iv: "062e96af761befb586d0e19a436287c8"
server_key_version: "encryptionkey@0"
I20260926 14:54:25.410645 2241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "502ab9218dce46febcb84279ae752800"
format_stamp: "Formatted at 2026-09-26 14:54:25 on dist-test-slave-24mn"
server_key: "f7a3495127096046730ef966c4bac1b1"
server_key_iv: "062e96af761befb586d0e19a436287c8"
server_key_version: "encryptionkey@0"
I20260926 14:54:25.414044 2241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:54:25.416363 2256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:25.417513 2241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:54:25.417624 2241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "502ab9218dce46febcb84279ae752800"
format_stamp: "Formatted at 2026-09-26 14:54:25 on dist-test-slave-24mn"
server_key: "f7a3495127096046730ef966c4bac1b1"
server_key_iv: "062e96af761befb586d0e19a436287c8"
server_key_version: "encryptionkey@0"
I20260926 14:54:25.417708 2241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:25.421896 2241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:25.424850 2241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:25.425005 2241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:25.431219 2241 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:35563
I20260926 14:54:25.431233 2308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:35563 every 8 connection(s)
I20260926 14:54:25.432114 2241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:25.434263 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 2241
I20260926 14:54:25.434381 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:25.434615 12563 external_mini_cluster.cc:1428] Setting key dd89637b0d234a6c5924d34cee90eb9b
I20260926 14:54:25.434675 2309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:25.439630 2312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:25.435775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56987 (local address 127.12.68.254:35563)
0926 14:54:25.436170 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:25.436186 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:25.436219 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:54:25.437130 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:25.437147 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:25.437410 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:25.437714 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:25.438518 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:25.438523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:25.438535 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:25.438546 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:25.438608 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:25.438630 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:25.438732 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:25.438839 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 14:54:25.438982 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":137,"threads_started":1}
I20260926 14:54:25.441074 2309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800: Bootstrap starting.
I20260926 14:54:25.442387 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:25.443420 2309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:25.444275 2309 log.cc:824] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:25.446076 2309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800: No bootstrap required, opened a new log
I20260926 14:54:25.446106 2312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:25.444483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50953 (local address 127.12.68.254:35563)
0926 14:54:25.444641 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:25.444645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:25.444938 (+ 293us) server_negotiation.cc:407] Connection header received
0926 14:54:25.445052 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:25.445055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:25.445084 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:25.445144 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:25.445764 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:25.445768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:25.445770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:54:25.445781 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:25.445805 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:54:25.445815 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:25.445880 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:25.445955 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 14:54:25.445989 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 14:54:25.449505 2309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502ab9218dce46febcb84279ae752800" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35563 } }
I20260926 14:54:25.449735 2309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:25.449802 2309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ab9218dce46febcb84279ae752800, State: Initialized, Role: FOLLOWER
I20260926 14:54:25.450197 2309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502ab9218dce46febcb84279ae752800" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35563 } }
I20260926 14:54:25.450316 2309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:25.450382 2309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:25.450474 2309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:25.451228 2309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502ab9218dce46febcb84279ae752800" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35563 } }
I20260926 14:54:25.451543 2309 leader_election.cc:302] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 502ab9218dce46febcb84279ae752800; no voters:
I20260926 14:54:25.451817 2309 leader_election.cc:288] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:25.451944 2319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:25.452116 2319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [term 1 LEADER]: Becoming Leader. State: Replica: 502ab9218dce46febcb84279ae752800, State: Running, Role: LEADER
I20260926 14:54:25.452461 2319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502ab9218dce46febcb84279ae752800" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35563 } }
I20260926 14:54:25.452862 2309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:25.453907 2321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 502ab9218dce46febcb84279ae752800. Latest consensus state: current_term: 1 leader_uuid: "502ab9218dce46febcb84279ae752800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502ab9218dce46febcb84279ae752800" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35563 } } }
I20260926 14:54:25.454027 2321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:25.454201 2320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "502ab9218dce46febcb84279ae752800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502ab9218dce46febcb84279ae752800" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 35563 } } }
I20260926 14:54:25.454342 2320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:25.454403 2328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:25.456979 2328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:25.461862 2328 catalog_manager.cc:1347] Generated new cluster ID: 4220af0af3cf485d9edc066d5f9ec751
I20260926 14:54:25.461922 2328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:25.484323 2328 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:25.484603 2328 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:25.488335 2328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 502ab9218dce46febcb84279ae752800: Generated new TSK 0
I20260926 14:54:25.488914 2328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:54:25.492936 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 2241
2026-09-26T14:54:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1273 ms)
[----------] 18 tests from AuthTokenIssuingTest (23284 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:25 dist-test-slave-24mn krb5kdc[2340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:25 dist-test-slave-24mn krb5kdc[2340](info): set up 2 sockets
Sep 26 14:54:25 dist-test-slave-24mn krb5kdc[2340](info): commencing operation
krb5kdc: starting...
W20260926 14:54:26.443809 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:26 dist-test-slave-24mn krb5kdc[2340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:54:26.552992 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:36075
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:36075,127.12.68.253:44649,127.12.68.252:33691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:26.655649 2353 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:54:26.655926 2353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:26.655968 2353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:26.656021 2353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:26.659832 2353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:26.659902 2353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:26.659931 2353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:26.659960 2353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:26.659982 2353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:26.664958 2353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:36075,127.12.68.253:44649,127.12.68.252:33691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:36075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:26.665896 2353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:26.666822 2353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:54:26.674836 2353 server_base.cc:1034] running on GCE node
W20260926 14:54:26.674886 2361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:26.674911 2359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:26.674898 2358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:26.675545 2353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:26.676601 2353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:26.677766 2353 hybrid_clock.cc:648] HybridClock initialized: now 1790434466677750 us; error 28 us; skew 500 ppm
Sep 26 14:54:26 dist-test-slave-24mn krb5kdc[2340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:26.680295 2353 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:54:26.681074 2353 webserver.cc:460] Webserver started at http://127.12.68.254:35773/ using document root <none> and password file <none>
I20260926 14:54:26.681548 2353 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:26.681602 2353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:26.681746 2353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:26.683665 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "c1ed08861c3f45a9be939215bfd9ca88"
format_stamp: "Formatted at 2026-09-26 14:54:26 on dist-test-slave-24mn"
server_key: "d8537103f90fd0830e6e6db38d0ec324"
server_key_iv: "b65a3a73007e6e1db449298072c0d919"
server_key_version: "encryptionkey@0"
I20260926 14:54:26.684180 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "c1ed08861c3f45a9be939215bfd9ca88"
format_stamp: "Formatted at 2026-09-26 14:54:26 on dist-test-slave-24mn"
server_key: "d8537103f90fd0830e6e6db38d0ec324"
server_key_iv: "b65a3a73007e6e1db449298072c0d919"
server_key_version: "encryptionkey@0"
I20260926 14:54:26.687425 2353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:54:26.689567 2368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:26.690692 2353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:54:26.690804 2353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "c1ed08861c3f45a9be939215bfd9ca88"
format_stamp: "Formatted at 2026-09-26 14:54:26 on dist-test-slave-24mn"
server_key: "d8537103f90fd0830e6e6db38d0ec324"
server_key_iv: "b65a3a73007e6e1db449298072c0d919"
server_key_version: "encryptionkey@0"
I20260926 14:54:26.690891 2353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:26.715195 2353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:26.718102 2353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:26.718264 2353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:26.724419 2353 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:36075
I20260926 14:54:26.724654 2420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:36075 every 8 connection(s)
I20260926 14:54:26.725291 2353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:26.727802 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:26.727990 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 2353
I20260926 14:54:26.728112 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:26.728391 12563 external_mini_cluster.cc:1428] Setting key f2795b29d325faa924444799a724e90e
I20260926 14:54:26.731748 2421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.68.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.253@KRBTEST.COM" created.
Sep 26 14:54:26 dist-test-slave-24mn krb5kdc[2340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:54:26.746264 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.734305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36075 (local address 127.0.0.1:57062)
0926 14:54:26.734756 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:26.734794 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:26.735607 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:26.736305 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:26.736310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:26.736726 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:26.736955 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:26.736967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.738056 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.738060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:26.739096 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:26.739108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.739759 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.739763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:26.739871 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:26.740386 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:26.740406 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:26.741845 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:26.743799 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:26.743806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:26.743812 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:26.744056 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:26.744374 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:26.744375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:26.744377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:26.744467 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:26.745100 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:26.745105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:26.745207 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 14:54:26.745325 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 14:54:26.745573 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":133,"threads_started":1}
I20260926 14:54:26.746265 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.735220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.12.68.254:36075)
0926 14:54:26.735530 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:26.735542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:26.735583 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:54:26.735949 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:26.735954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:26.736137 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:26.736295 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:26.737096 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.737943 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.739223 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.739609 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.739733 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:26.741986 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:26.741992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:26.742004 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:26.742029 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:26.743686 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:26.744156 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:26.744162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:26.744164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:26.744292 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:26.744550 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:26.744554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:26.744556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:26.745004 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:26.745142 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:26.745430 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 14:54:26.745663 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
I20260926 14:54:26.748522 2421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:54:26.749073 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.748877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44649 (local address 127.0.0.1:50648)
0926 14:54:26.748964 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:26.749025 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:54:26.749228 2369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.68.253:44649: connect: Connection refused (error 111)
I20260926 14:54:26.749936 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.749759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44649 (local address 127.0.0.1:50652)
0926 14:54:26.749852 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:26.749892 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:54:26.750046 2421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.253:44649: Network error: Client connection negotiation failed: client connection to 127.12.68.253:44649: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.253.keytab.
Entry for principal kudu/127.12.68.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.253.keytab.
I20260926 14:54:26.754339 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.253:44649
--webserver_interface=127.12.68.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.253.keytab
--principal=kudu/127.12.68.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:36075,127.12.68.253:44649,127.12.68.252:33691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:54:26.801025 2421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } attempt: 1
I20260926 14:54:26.801837 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.801532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44649 (local address 127.0.0.1:50654)
0926 14:54:26.801674 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:26.801743 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 14:54:26.803375 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.802503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44649 (local address 127.0.0.1:50664)
0926 14:54:26.803257 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:26.803312 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":678}
W20260926 14:54:26.803572 2421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.253:44649: Network error: Client connection negotiation failed: client connection to 127.12.68.253:44649: connect: Connection refused (error 111)
W20260926 14:54:26.861604 2428 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:54:26.861848 2428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:26.861869 2428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:26.861909 2428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:26.865741 2428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:26.865788 2428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:26.865808 2428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:26.865821 2428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:26.865834 2428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:26.870762 2428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:36075,127.12.68.253:44649,127.12.68.252:33691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.253.keytab
--principal=kudu/127.12.68.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.253:44649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.68.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:26.871752 2421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } attempt: 2
I20260926 14:54:26.871824 2428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:26.872536 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.872269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44649 (local address 127.0.0.1:50678)
0926 14:54:26.872406 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:26.872465 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 14:54:26.872902 2428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:54:26.873240 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.873003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44649 (local address 127.0.0.1:50694)
0926 14:54:26.873115 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:26.873172 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:54:26.873390 2421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.253:44649: Network error: Client connection negotiation failed: client connection to 127.12.68.253:44649: connect: Connection refused (error 111)
W20260926 14:54:26.880798 2436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:26.880851 2428 server_base.cc:1034] running on GCE node
W20260926 14:54:26.880795 2434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:26.880975 2433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:26.881488 2428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:26.882530 2428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:26.883684 2428 hybrid_clock.cc:648] HybridClock initialized: now 1790434466883661 us; error 33 us; skew 500 ppm
Sep 26 14:54:26 dist-test-slave-24mn krb5kdc[2340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:26.886096 2428 init.cc:377] Logged in from keytab as kudu/127.12.68.253@KRBTEST.COM (short username kudu)
I20260926 14:54:26.886868 2428 webserver.cc:460] Webserver started at http://127.12.68.253:36407/ using document root <none> and password file <none>
I20260926 14:54:26.887346 2428 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:26.887408 2428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:26.887588 2428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:26.889495 2428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/data/instance:
uuid: "ffd2ca8bd8764ec6bb039278764b90b8"
format_stamp: "Formatted at 2026-09-26 14:54:26 on dist-test-slave-24mn"
server_key: "d375e0113a9e0483d103d93a55787118"
server_key_iv: "aa2044279ca786afa39c8da144aa1bfb"
server_key_version: "encryptionkey@0"
I20260926 14:54:26.889995 2428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/wal/instance:
uuid: "ffd2ca8bd8764ec6bb039278764b90b8"
format_stamp: "Formatted at 2026-09-26 14:54:26 on dist-test-slave-24mn"
server_key: "d375e0113a9e0483d103d93a55787118"
server_key_iv: "aa2044279ca786afa39c8da144aa1bfb"
server_key_version: "encryptionkey@0"
I20260926 14:54:26.893451 2428 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:54:26.895731 2443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:26.896862 2428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:54:26.896975 2428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/wal
uuid: "ffd2ca8bd8764ec6bb039278764b90b8"
format_stamp: "Formatted at 2026-09-26 14:54:26 on dist-test-slave-24mn"
server_key: "d375e0113a9e0483d103d93a55787118"
server_key_iv: "aa2044279ca786afa39c8da144aa1bfb"
server_key_version: "encryptionkey@0"
I20260926 14:54:26.897063 2428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:26.909736 2428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:26.912243 2428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:26.912400 2428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:26.918459 2428 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.253:44649
I20260926 14:54:26.918686 2495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.253:44649 every 8 connection(s)
I20260926 14:54:26.919325 2428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/data/info.pb
I20260926 14:54:26.921953 2496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:26.923003 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 2428
I20260926 14:54:26.923115 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-1/wal/instance
I20260926 14:54:26.923374 12563 external_mini_cluster.cc:1428] Setting key f95fca3b10b42ea9fb29f3107f525b32
I20260926 14:54:26.925863 2496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.68.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.252@KRBTEST.COM" created.
Sep 26 14:54:26 dist-test-slave-24mn krb5kdc[2340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.253@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:54:26.938544 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.927569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57066 (local address 127.12.68.254:36075)
0926 14:54:26.927722 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:26.927727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:26.928484 (+ 757us) server_negotiation.cc:407] Connection header received
0926 14:54:26.929379 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:26.929383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:26.929440 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:26.929540 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:26.930587 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.931108 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.932296 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.932644 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.932754 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:26.935155 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:26.935164 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:26.935167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:26.935193 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:26.936680 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:26.937123 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:26.937127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:26.937129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:26.937220 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:26.937493 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:26.937496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:26.937498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:26.937722 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:26.937825 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:26.938248 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 14:54:26.938374 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:54:26.939162 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.927875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36075 (local address 127.0.0.1:57066)
0926 14:54:26.928337 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:26.928384 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:26.929177 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:26.929665 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:26.929671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:26.929970 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:26.930452 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:26.930468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.931215 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.931218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:26.932183 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:26.932192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.932802 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.932806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:26.932998 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:26.933565 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:26.933585 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:26.934995 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:26.936782 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:26.936786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:26.936796 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:26.937025 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:26.937299 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:26.937301 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:26.937303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:26.937414 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:26.937808 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:26.937812 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:26.938019 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 14:54:26.938168 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 14:54:26.938387 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":105,"threads_started":1}
I20260926 14:54:26.940470 2496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:54:26 dist-test-slave-24mn krb5kdc[2340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.253@KRBTEST.COM for kudu/127.12.68.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.252.keytab.
Entry for principal kudu/127.12.68.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.252.keytab.
I20260926 14:54:26.948398 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.940929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44649 (local address 127.0.0.1:50700)
0926 14:54:26.941046 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:26.941060 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:26.941119 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:26.941615 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:26.941618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:26.941742 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:26.941830 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:26.941835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.942714 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.942716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:26.943254 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:26.943263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.943826 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.943828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:26.943928 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:26.944314 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:26.944324 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:26.945433 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:26.946953 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:26.946956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:26.946957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:26.947137 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:26.947362 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:26.947364 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:26.947365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:26.947426 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:26.947867 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:26.947870 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:26.947983 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 14:54:26.948098 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 14:54:26.948207 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 14:54:26.948374 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.252:33691
--webserver_interface=127.12.68.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.252.keytab
--principal=kudu/127.12.68.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:36075,127.12.68.253:44649,127.12.68.252:33691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:54:26.948426 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.941009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.12.68.253:44649)
0926 14:54:26.941267 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:26.941275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:26.941294 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:54:26.941333 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:26.941338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:26.941499 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:26.941613 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:26.941918 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.942617 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.943343 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.943740 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.943816 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:26.945558 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:26.945566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:26.945575 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:26.945595 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:26.946855 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:26.947219 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:26.947223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:26.947224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:26.947293 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:26.947493 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:26.947496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:26.947498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:26.947772 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:26.947872 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:26.948160 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 14:54:26.948267 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":95,"threads_started":1}
I20260926 14:54:26.950001 2496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:54:26.950593 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.950381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56382)
0926 14:54:26.950467 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:26.950543 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 14:54:26.950744 2444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
I20260926 14:54:26.951464 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.951260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56394)
0926 14:54:26.951362 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:26.951409 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:54:26.951577 2496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:33691: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
I20260926 14:54:26.964545 2421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } attempt: 3
Sep 26 14:54:26 dist-test-slave-24mn krb5kdc[2340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for kudu/127.12.68.253@KRBTEST.COM
I20260926 14:54:26.972635 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.965069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.12.68.253:44649)
0926 14:54:26.965194 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:26.965198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:26.965215 (+ 17us) server_negotiation.cc:407] Connection header received
0926 14:54:26.965351 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:26.965354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:26.965395 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:26.965457 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:26.965862 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.966381 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.967517 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.967761 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.967844 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:26.969917 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:26.969920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:26.969922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:26.969943 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:26.971139 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:26.971583 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:26.971585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:26.971586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:26.971659 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:26.971938 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:26.971941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:26.971944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:26.972110 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:26.972181 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:26.972439 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 14:54:26.972528 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 14:54:26.972687 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.965021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44649 (local address 127.0.0.1:50704)
0926 14:54:26.965155 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:26.965170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:26.965265 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:26.965484 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:26.965487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:26.965653 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:26.965757 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:26.965763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.966480 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.966483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:26.967405 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:26.967414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:26.967860 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:26.967864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:26.967987 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:26.968432 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:26.968444 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:26.969799 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:26.971239 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:26.971243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:26.971245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:26.971494 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:26.971738 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:26.971741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:26.971743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:26.971831 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:26.972193 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:26.972196 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:26.972298 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 14:54:26.972396 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 14:54:26.972525 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 14:54:26.973649 2421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:54:26.974256 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.974050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56398)
0926 14:54:26.974156 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:26.974200 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 14:54:26.974858 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:26.974650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56412)
0926 14:54:26.974768 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:26.974808 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 14:54:26.974962 2421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:33691: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
I20260926 14:54:27.000777 2496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } attempt: 1
I20260926 14:54:27.001567 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.001274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56424)
0926 14:54:27.001402 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:27.001467 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 14:54:27.002197 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.001996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56426)
0926 14:54:27.002094 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:27.002143 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 14:54:27.002303 2496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:33691: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
I20260926 14:54:27.006104 2421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } attempt: 1
I20260926 14:54:27.006699 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.006473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56442)
0926 14:54:27.006590 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:27.006640 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 14:54:27.007323 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.007155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56444)
0926 14:54:27.007237 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:27.007277 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 14:54:27.007457 2421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:33691: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
W20260926 14:54:27.055737 2502 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:54:27.056039 2502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:27.056082 2502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:27.056134 2502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:27.060089 2502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:27.060135 2502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:27.060153 2502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:27.060168 2502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:27.060182 2502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:27.065281 2502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:36075,127.12.68.253:44649,127.12.68.252:33691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.252.keytab
--principal=kudu/127.12.68.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.252:33691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.68.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:27.066264 2502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:27.067214 2502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:54:27.070461 2496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } attempt: 2
I20260926 14:54:27.071337 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.070986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56458)
0926 14:54:27.071161 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:27.071233 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 14:54:27.072077 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.071861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56472)
0926 14:54:27.071979 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:27.072027 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 14:54:27.072184 2496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:33691: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
W20260926 14:54:27.075385 2508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:27.075409 2511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:27.075409 2509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:27.075594 2502 server_base.cc:1034] running on GCE node
I20260926 14:54:27.076154 2502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:27.077149 2502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:27.078307 2502 hybrid_clock.cc:648] HybridClock initialized: now 1790434467078276 us; error 44 us; skew 500 ppm
Sep 26 14:54:27 dist-test-slave-24mn krb5kdc[2340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434467, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:27.080901 2502 init.cc:377] Logged in from keytab as kudu/127.12.68.252@KRBTEST.COM (short username kudu)
I20260926 14:54:27.081725 2502 webserver.cc:460] Webserver started at http://127.12.68.252:41113/ using document root <none> and password file <none>
I20260926 14:54:27.082230 2502 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:27.082290 2502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:27.082463 2502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:27.082592 2421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } attempt: 2
I20260926 14:54:27.083330 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.083050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56474)
0926 14:54:27.083194 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:27.083255 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 14:54:27.083978 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.083754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56478)
0926 14:54:27.083848 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:27.083919 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 14:54:27.084092 2421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:33691: Network error: Client connection negotiation failed: client connection to 127.12.68.252:33691: connect: Connection refused (error 111)
I20260926 14:54:27.084759 2502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/data/instance:
uuid: "06ce64299ecb407ba5da14f93e9852f0"
format_stamp: "Formatted at 2026-09-26 14:54:27 on dist-test-slave-24mn"
server_key: "c3269d3fbf75350df3e6d04f04d83592"
server_key_iv: "9aaa38caca8fc4e2143379d8d1d40aae"
server_key_version: "encryptionkey@0"
I20260926 14:54:27.085237 2502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/wal/instance:
uuid: "06ce64299ecb407ba5da14f93e9852f0"
format_stamp: "Formatted at 2026-09-26 14:54:27 on dist-test-slave-24mn"
server_key: "c3269d3fbf75350df3e6d04f04d83592"
server_key_iv: "9aaa38caca8fc4e2143379d8d1d40aae"
server_key_version: "encryptionkey@0"
I20260926 14:54:27.088578 2502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:54:27.090668 2518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:27.091838 2502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:27.091965 2502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/wal
uuid: "06ce64299ecb407ba5da14f93e9852f0"
format_stamp: "Formatted at 2026-09-26 14:54:27 on dist-test-slave-24mn"
server_key: "c3269d3fbf75350df3e6d04f04d83592"
server_key_iv: "9aaa38caca8fc4e2143379d8d1d40aae"
server_key_version: "encryptionkey@0"
I20260926 14:54:27.092072 2502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:27.104874 2502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:27.107488 2502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:27.107647 2502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:27.114192 2502 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.252:33691
I20260926 14:54:27.114450 2570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.252:33691 every 8 connection(s)
I20260926 14:54:27.115089 2502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/data/info.pb
I20260926 14:54:27.117664 2571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:27.121264 2571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:54:27.123842 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 2502
I20260926 14:54:27.123996 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434204424021-12563-0/minicluster-data/master-2/wal/instance
I20260926 14:54:27.124300 12563 external_mini_cluster.cc:1428] Setting key e90cb715955f1f27d9ccfa652ef21fb8
Sep 26 14:54:27 dist-test-slave-24mn krb5kdc[2340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434467, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.252@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
Sep 26 14:54:27 dist-test-slave-24mn krb5kdc[2340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.252@KRBTEST.COM
I20260926 14:54:27.135066 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.123038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.12.68.254:36075)
0926 14:54:27.123172 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.123176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.124050 (+ 874us) server_negotiation.cc:407] Connection header received
0926 14:54:27.125150 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.125154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.125211 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.125263 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.126422 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.127199 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.128534 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.128780 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.128880 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.131393 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:27.131397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:27.131399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:27.131426 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:27.132863 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.133453 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.133455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.133456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.133524 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.133830 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.133834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.133836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.134058 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:27.134153 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:27.134773 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 14:54:27.134890 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:54:27.136016 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.123375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:36075 (local address 127.0.0.1:57076)
0926 14:54:27.123853 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:27.123940 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:27.124922 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:27.125521 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:27.125528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:27.125927 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:27.126272 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:27.126291 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.127350 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.127353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:27.128420 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:27.128431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.128913 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.128916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:27.129124 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.129696 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:27.129717 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:27.131241 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:27.132973 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:27.132980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:27.132989 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:27.133358 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:27.133599 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:27.133602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:27.133604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:27.133744 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:27.134228 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:27.134234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:27.134539 (+ 305us) client_negotiation.cc:769] Sending connection context
0926 14:54:27.134703 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 14:54:27.135016 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":42,"spinlock_wait_cycles":1920,"thread_start_us":111,"threads_started":1}
I20260926 14:54:27.136020 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.124906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.12.68.252:33691)
0926 14:54:27.125203 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.125211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.125243 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:54:27.125422 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.125426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.125592 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.125720 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.126532 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.127618 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.128731 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.129206 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.129279 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.131755 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:27.131760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:27.131770 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:27.131789 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:27.133515 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.133884 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.133889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.133890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.133956 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.134224 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.134228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.134230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.134522 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:27.134723 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:27.134875 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 14:54:27.135074 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":217,"thread_start_us":105,"threads_started":1}
I20260926 14:54:27.137784 2571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:54:27 dist-test-slave-24mn krb5kdc[2340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434467, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.252@KRBTEST.COM for kudu/127.12.68.253@KRBTEST.COM
Sep 26 14:54:27 dist-test-slave-24mn krb5kdc[2340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:54:27.145447 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.138215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:44649 (local address 127.0.0.1:50714)
0926 14:54:27.138305 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:27.138318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:27.138388 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:27.138640 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:27.138642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:27.138798 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:27.138910 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:27.138917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.139786 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.139789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:27.140570 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:27.140580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.140999 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.141002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:27.141076 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.141459 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:27.141473 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:27.142657 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:27.144162 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:27.144165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:27.144167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:27.144343 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:27.144578 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:27.144581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:27.144583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:27.144657 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:27.145011 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:27.145015 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:27.145100 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 14:54:27.145183 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 14:54:27.145298 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 14:54:27.145473 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.138302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.12.68.253:44649)
0926 14:54:27.138430 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.138434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.138447 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:54:27.138488 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.138491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.138538 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.138613 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.139022 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.139686 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.140676 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.140904 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.140994 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.142783 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:27.142786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:27.142787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:27.142809 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:27.144071 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.144426 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.144429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.144431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.144500 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.144731 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.144734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.144736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.144924 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:27.145007 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:27.145243 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 14:54:27.145343 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:54:27.146060 2571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:54:27.146195 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.138187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.12.68.254:36075)
0926 14:54:27.138289 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.138293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.138478 (+ 185us) server_negotiation.cc:407] Connection header received
0926 14:54:27.138574 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.138577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.138617 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.138678 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.139281 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.139939 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.140863 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.141133 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.141204 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.143299 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:27.143301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:27.143303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:27.143323 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:27.144610 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.144966 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.144969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.144971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.145042 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.145271 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.145274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.145276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.145445 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:27.145654 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:27.145956 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 14:54:27.146065 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 14:54:27 dist-test-slave-24mn krb5kdc[2340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434467, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.252@KRBTEST.COM for kudu/127.12.68.252@KRBTEST.COM
Sep 26 14:54:27 dist-test-slave-24mn krb5kdc[2340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.253@KRBTEST.COM
I20260926 14:54:27.153801 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.146452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56506)
0926 14:54:27.146548 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:27.146561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:27.146626 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:27.146865 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:27.146867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:27.147000 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:27.147086 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:27.147092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.147780 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.147784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:27.148424 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:27.148433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.148966 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.148971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:27.149054 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.149429 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:27.149441 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:27.150782 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:27.152467 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:27.152470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:27.152472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:27.152637 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:27.152877 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:27.152880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:27.152882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:27.152964 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:27.153315 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:27.153318 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:27.153404 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 14:54:27.153508 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 14:54:27.153632 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 14:54:27.153810 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.146548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.12.68.252:33691)
0926 14:54:27.146626 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.146630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.146642 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:54:27.146692 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.146694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.146725 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.146784 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.147189 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.147682 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.148531 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.148855 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.148924 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.150927 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:27.150930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:27.150933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:27.150968 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:27.152376 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.152714 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.152717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.152719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.152795 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.153063 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.153065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.153066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.153225 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:27.153318 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:27.153556 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 14:54:27.153683 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":27}
I20260926 14:54:27.155601 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.147220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50728 (local address 127.12.68.253:44649)
0926 14:54:27.147316 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.147320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.147331 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:54:27.147418 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.147421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.147456 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.147522 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.148161 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.148674 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.149651 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.149872 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.149953 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.152550 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:27.152553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:27.152555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:27.152576 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:27.154113 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.154524 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.154527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.154529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.154607 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.154865 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.154868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.154870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.155078 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:27.155287 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:27.155353 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 14:54:27.155454 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:54:27.157483 2571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0: Bootstrap starting.
I20260926 14:54:27.158707 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:27.160216 2571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:27.160925 2571 log.cc:824] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:27.160112 2395 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c1ed08861c3f45a9be939215bfd9ca88"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57092
I20260926 14:54:27.162806 2571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0: No bootstrap required, opened a new log
I20260926 14:54:27.163329 2496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } attempt: 3
I20260926 14:54:27.166704 2571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } }
I20260926 14:54:27.166939 2571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ce64299ecb407ba5da14f93e9852f0, State: Initialized, Role: FOLLOWER
I20260926 14:54:27.167274 2571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } }
Sep 26 14:54:27 dist-test-slave-24mn krb5kdc[2340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.253@KRBTEST.COM for kudu/127.12.68.252@KRBTEST.COM
I20260926 14:54:27.167730 2577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } } }
I20260926 14:54:27.167872 2577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:54:27.168628 2571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:27.171155 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.163708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56516)
0926 14:54:27.163831 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:27.163847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:27.163960 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:27.164152 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:27.164155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:27.164340 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:27.164442 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:27.164449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.165126 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.165129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:27.165635 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:27.165643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.166058 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.166061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:27.166137 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.166537 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:27.166547 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:27.167772 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:27.169633 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:27.169637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:27.169639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:27.169872 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:27.170148 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:27.170151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:27.170153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:27.170235 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:27.170683 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:27.170686 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:27.170777 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 14:54:27.170871 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 14:54:27.170998 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 14:54:27.171155 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.163759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56516 (local address 127.12.68.252:33691)
0926 14:54:27.163854 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.163858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.163907 (+ 49us) server_negotiation.cc:407] Connection header received
0926 14:54:27.164043 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.164045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.164078 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.164133 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.164557 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.165027 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.165735 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.165965 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.166048 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.167955 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:27.167959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:27.167961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:27.167995 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:27.169514 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.169975 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.169978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.169980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.170065 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.170317 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.170320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.170322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.170553 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:27.170639 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:27.170925 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 14:54:27.171018 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 14:54:27.174355 2496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8: Bootstrap starting.
I20260926 14:54:27.172362 2502 master_runner.cc:186] Error getting master registration for 127.12.68.254:36075: OK, instance_id { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" instance_seqno: 1790434466722769 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:54:27.176090 2502 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:54:27.176883 2496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:27.177573 2496 log.cc:824] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:27.179291 2496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8: No bootstrap required, opened a new log
W20260926 14:54:27.181031 2588 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:54:27.181103 2588 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:54:27.182533 2496 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } }
I20260926 14:54:27.182703 2496 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd2ca8bd8764ec6bb039278764b90b8, State: Initialized, Role: FOLLOWER
I20260926 14:54:27.183008 2496 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } }
I20260926 14:54:27.183244 2421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } attempt: 3
I20260926 14:54:27.184211 2496 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:27.184302 2591 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } } }
I20260926 14:54:27.184561 2591 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 14:54:27 dist-test-slave-24mn krb5kdc[2340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for kudu/127.12.68.252@KRBTEST.COM
W20260926 14:54:27.188966 2602 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:54:27.189117 2602 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:54:27.187323 2428 master_runner.cc:186] Error getting master registration for 127.12.68.254:36075: OK, instance_id { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" instance_seqno: 1790434466722769 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:54:27.190925 2428 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:54:27.193007 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.183670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:33691 (local address 127.0.0.1:56518)
0926 14:54:27.183981 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:27.183995 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:27.184078 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:27.184393 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:27.184397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:27.184589 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:27.184700 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:27.184708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.185671 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.185675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:27.186503 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:27.186514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.187182 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.187198 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:27.187287 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.187846 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:27.187860 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:27.189318 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:27.191374 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:27.191377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:27.191379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:27.191634 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:27.191944 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:27.191947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:27.191949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:27.192059 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:27.192478 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:27.192481 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:27.192604 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 14:54:27.192709 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 14:54:27.192846 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260926 14:54:27.193033 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.183997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56518 (local address 127.12.68.252:33691)
0926 14:54:27.184094 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.184099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.184113 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:54:27.184150 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.184153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.184194 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.184269 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.184822 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.185561 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.186637 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.186970 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.187054 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.189507 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:27.189510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:27.189513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:27.189537 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:27.191248 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.191735 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.191738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.191740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.191822 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.192156 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.192159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.192161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.192373 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:27.192464 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:27.192778 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 14:54:27.192886 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:54:27.196120 2421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88: Bootstrap starting.
I20260926 14:54:27.198807 2421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:27.199807 2421 log.cc:824] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:27.202188 2421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88: No bootstrap required, opened a new log
I20260926 14:54:27.205145 2421 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } }
I20260926 14:54:27.205371 2421 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ed08861c3f45a9be939215bfd9ca88, State: Initialized, Role: FOLLOWER
I20260926 14:54:27.205693 2421 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } }
I20260926 14:54:27.206209 2605 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } } }
I20260926 14:54:27.206477 2605 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:54:27.206950 2421 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:27.207017 2395 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 14:54:27.207116 2395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:27.208294 2395 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } }
I20260926 14:54:27.209173 2395 leader_election.cc:288] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [CANDIDATE]: Term 1 election: Requested vote from peers ffd2ca8bd8764ec6bb039278764b90b8 (127.12.68.253:44649), 06ce64299ecb407ba5da14f93e9852f0 (127.12.68.252:33691)
I20260926 14:54:27.209946 2470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1ed08861c3f45a9be939215bfd9ca88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ffd2ca8bd8764ec6bb039278764b90b8"
I20260926 14:54:27.210186 2470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:27.211437 2470 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1ed08861c3f45a9be939215bfd9ca88 in term 1.
I20260926 14:54:27.211875 2545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1ed08861c3f45a9be939215bfd9ca88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "06ce64299ecb407ba5da14f93e9852f0"
I20260926 14:54:27.212150 2545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:27.213349 2545 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1ed08861c3f45a9be939215bfd9ca88 in term 1.
I20260926 14:54:27.213758 2369 leader_election.cc:302] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ce64299ecb407ba5da14f93e9852f0, c1ed08861c3f45a9be939215bfd9ca88; no voters:
I20260926 14:54:27.214071 2605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:27.214371 2605 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [term 1 LEADER]: Becoming Leader. State: Replica: c1ed08861c3f45a9be939215bfd9ca88, State: Running, Role: LEADER
I20260926 14:54:27.214753 2605 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } }
I20260926 14:54:27.216595 2605 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1ed08861c3f45a9be939215bfd9ca88. Latest consensus state: current_term: 1 leader_uuid: "c1ed08861c3f45a9be939215bfd9ca88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } } }
I20260926 14:54:27.216756 2605 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:27.218487 2623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:27.218812 2353 master_runner.cc:263] No leader master found from master c1ed08861c3f45a9be939215bfd9ca88
I20260926 14:54:27.218881 2353 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 14:54:27.219650 2374 tablet.cc:2369] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:54:27.222807 2623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:27.242452 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.220173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.12.68.254:36075)
0926 14:54:27.220282 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.220286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.228204 (+ 7918us) server_negotiation.cc:407] Connection header received
0926 14:54:27.228414 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.228418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.228471 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.228522 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.229504 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.230260 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.231457 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.231816 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.231911 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.235935 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:27.235938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:27.235941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:27.235968 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:27.237661 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.240187 (+ 2526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.240190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.240192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.240283 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.240572 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.240575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.240577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.240781 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:27.240881 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:27.242148 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0926 14:54:27.242275 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:54:27.244585 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.220518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50744 (local address 127.12.68.253:44649)
0926 14:54:27.220618 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.220622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.229434 (+ 8812us) server_negotiation.cc:407] Connection header received
0926 14:54:27.229602 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.229605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.229654 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.229704 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.230583 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.231315 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.232823 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.233154 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.233231 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.238150 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:27.238154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:27.238157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:27.238186 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:27.239710 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.240952 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.240955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.240957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.241038 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.241537 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.241541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.241543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.241741 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:27.241838 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:27.244297 (+ 2459us) server_negotiation.cc:299] Negotiation successful
0926 14:54:27.244431 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:54:27.249011 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:27.223177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.12.68.252:33691)
0926 14:54:27.223278 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.223283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.232366 (+ 9083us) server_negotiation.cc:407] Connection header received
0926 14:54:27.232535 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.232539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.232592 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.232641 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.233515 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.234232 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.235280 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:27.235634 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:27.235707 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:27.244823 (+ 9116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:27.244826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:27.244829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:27.244856 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:27.246335 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.246960 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.246963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.246965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.247046 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:27.247319 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:27.247322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:27.247323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:27.247533 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:27.247625 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:27.248744 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0926 14:54:27.248858 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:54:27.249320 2545 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0 [term 1 FOLLOWER]: Refusing update from remote peer c1ed08861c3f45a9be939215bfd9ca88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:27.250113 2470 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8 [term 1 FOLLOWER]: Refusing update from remote peer c1ed08861c3f45a9be939215bfd9ca88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:27.250770 2605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:27.251631 2605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:27.257827 2577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1ed08861c3f45a9be939215bfd9ca88. Latest consensus state: current_term: 1 leader_uuid: "c1ed08861c3f45a9be939215bfd9ca88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } } }
I20260926 14:54:27.257827 2591 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1ed08861c3f45a9be939215bfd9ca88. Latest consensus state: current_term: 1 leader_uuid: "c1ed08861c3f45a9be939215bfd9ca88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } } }
I20260926 14:54:27.257949 2577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:54:27.257949 2591 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:54:27.263186 2605 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1ed08861c3f45a9be939215bfd9ca88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } } }
I20260926 14:54:27.263326 2605 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:27.264473 2577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1ed08861c3f45a9be939215bfd9ca88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } } }
I20260926 14:54:27.264583 2577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:54:27.268324 2630 mvcc.cc:205] Tried to move back new op lower bound from 7333619577843933184 to 7333619577715318784. Current Snapshot: MvccSnapshot[applied={T|T < 7333619577843933184}]
I20260926 14:54:27.269098 2619 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1ed08861c3f45a9be939215bfd9ca88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } } }
I20260926 14:54:27.269241 2619 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:27.270327 2629 mvcc.cc:205] Tried to move back new op lower bound from 7333619577843933184 to 7333619577715318784. Current Snapshot: MvccSnapshot[applied={T|T < 7333619577843933184}]
I20260926 14:54:27.270486 2591 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1ed08861c3f45a9be939215bfd9ca88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ed08861c3f45a9be939215bfd9ca88" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 36075 } } peers { permanent_uuid: "ffd2ca8bd8764ec6bb039278764b90b8" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 44649 } } peers { permanent_uuid: "06ce64299ecb407ba5da14f93e9852f0" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 33691 } } }
I20260926 14:54:27.270581 2591 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:54:27.276463 2623 catalog_manager.cc:1347] Generated new cluster ID: b3b406dac8c3409591d3ba461a41a189
I20260926 14:54:27.276533 2623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:27.315493 2623 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:27.316740 2623 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:27.325841 2623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1ed08861c3f45a9be939215bfd9ca88: Generated new TSK 0
I20260926 14:54:27.328838 2623 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:54:27.373286 2425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:27.368330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.12.68.254:36075)
0926 14:54:27.368454 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.368460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.372201 (+ 3741us) server_negotiation.cc:407] Connection header received
0926 14:54:27.372400 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.372404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.372455 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.372505 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.373155 (+ 650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57116: BlockingRecv error: recv got EOF from 127.0.0.1:57116 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 14:54:27.375036 2501 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:27.369125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.12.68.253:44649)
0926 14:54:27.369237 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.369241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.373683 (+ 4442us) server_negotiation.cc:407] Connection header received
0926 14:54:27.374271 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.374274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.374324 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.374373 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.374923 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50756: BlockingRecv error: recv got EOF from 127.0.0.1:50756 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 14:54:27.375990 2575 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:27.369802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.12.68.252:33691)
0926 14:54:27.369925 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.369929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.374014 (+ 4085us) server_negotiation.cc:407] Connection header received
0926 14:54:27.375274 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.375278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.375328 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.375376 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.375847 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56542: BlockingRecv error: recv got EOF from 127.0.0.1:56542 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 14:54:27.380414 2501 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:27.378908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.12.68.253:44649)
0926 14:54:27.379008 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.379011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.379466 (+ 455us) server_negotiation.cc:407] Connection header received
0926 14:54:27.379657 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.379660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.379727 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.379775 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.380321 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50770: BlockingRecv error: recv got EOF from 127.0.0.1:50770 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 14:54:27.386175 2501 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:27.383695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.12.68.253:44649)
0926 14:54:27.383788 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.383792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.385413 (+ 1621us) server_negotiation.cc:407] Connection header received
0926 14:54:27.385537 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.385540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.385585 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.385628 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.386087 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50782: BlockingRecv error: recv got EOF from 127.0.0.1:50782 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 14:54:27.390547 2501 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:27.389650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.12.68.253:44649)
0926 14:54:27.389743 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.389747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.389905 (+ 158us) server_negotiation.cc:407] Connection header received
0926 14:54:27.390003 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.390006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.390046 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.390105 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.390459 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50792: BlockingRecv error: recv got EOF from 127.0.0.1:50792 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 14:54:27.396924 2501 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:27.396007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.12.68.253:44649)
0926 14:54:27.396136 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.396140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.396281 (+ 141us) server_negotiation.cc:407] Connection header received
0926 14:54:27.396381 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.396383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.396424 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.396483 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.396841 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50798: BlockingRecv error: recv got EOF from 127.0.0.1:50798 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 14:54:27.407532 2501 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:27.406554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50808 (local address 127.12.68.253:44649)
0926 14:54:27.406655 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.406660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.406803 (+ 143us) server_negotiation.cc:407] Connection header received
0926 14:54:27.406922 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.406926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.406967 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.407030 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.407428 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50808: BlockingRecv error: recv got EOF from 127.0.0.1:50808 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 14:54:27.426293 2501 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:27.425281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.12.68.253:44649)
0926 14:54:27.425387 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.425391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.425551 (+ 160us) server_negotiation.cc:407] Connection header received
0926 14:54:27.425678 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.425681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.425722 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.425787 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.426191 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50818: BlockingRecv error: recv got EOF from 127.0.0.1:50818 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 14:54:27.460954 2501 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:27.460007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.12.68.253:44649)
0926 14:54:27.460125 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.460128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.460317 (+ 189us) server_negotiation.cc:407] Connection header received
0926 14:54:27.460380 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.460382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.460430 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.460508 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.460852 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50820: BlockingRecv error: recv got EOF from 127.0.0.1:50820 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 14:54:27.527933 2501 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:27.526910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.12.68.253:44649)
0926 14:54:27.527026 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.527029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.527185 (+ 156us) server_negotiation.cc:407] Connection header received
0926 14:54:27.527302 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.527304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.527348 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.527425 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.527815 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50836: BlockingRecv error: recv got EOF from 127.0.0.1:50836 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 14:54:27.658938 2501 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:27.657807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.12.68.253:44649)
0926 14:54:27.657932 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.657936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.658127 (+ 191us) server_negotiation.cc:407] Connection header received
0926 14:54:27.658233 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.658235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.658283 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.658367 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.658815 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50838: BlockingRecv error: recv got EOF from 127.0.0.1:50838 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 14:54:27.918218 2501 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:27.917155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.12.68.253:44649)
0926 14:54:27.917288 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:27.917293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:27.917341 (+ 48us) server_negotiation.cc:407] Connection header received
0926 14:54:27.917537 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:27.917541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:27.917590 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:27.917660 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:27.918101 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50842: BlockingRecv error: recv got EOF from 127.0.0.1:50842 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:54:28.182514 2588 catalog_manager.cc:1259] Loaded cluster ID: b3b406dac8c3409591d3ba461a41a189
I20260926 14:54:28.182590 2588 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0: loading cluster ID for follower catalog manager: success
I20260926 14:54:28.184320 2588 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0: acquiring CA information for follower catalog manager: success
I20260926 14:54:28.185585 2588 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 06ce64299ecb407ba5da14f93e9852f0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:54:28.190276 2602 catalog_manager.cc:1259] Loaded cluster ID: b3b406dac8c3409591d3ba461a41a189
I20260926 14:54:28.190341 2602 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8: loading cluster ID for follower catalog manager: success
I20260926 14:54:28.191934 2602 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8: acquiring CA information for follower catalog manager: success
I20260926 14:54:28.192986 2602 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ffd2ca8bd8764ec6bb039278764b90b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:54:28.436444 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:28.432342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.12.68.253:44649)
0926 14:54:28.432654 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:28.432659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:28.432675 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:54:28.432798 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:28.432801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:28.432924 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:28.433016 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:28.433296 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:28.433819 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:28.434970 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:28.435256 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:28.435358 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:28.435589 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:28.436097 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:28.436189 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:28.436238 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 14:54:28.436288 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":90,"threads_started":1}
I20260926 14:54:28.442775 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:28.439427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.12.68.252:33691)
0926 14:54:28.439650 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:28.439653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:28.439666 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:54:28.439739 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:28.439741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:28.439846 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:28.439963 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:28.440136 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:28.440644 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:28.441458 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:28.441706 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:28.441783 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:28.441994 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:28.442401 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:28.442489 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:28.442572 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 14:54:28.442620 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
I20260926 14:54:28.449644 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:28.446325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56568 (local address 127.12.68.252:33691)
0926 14:54:28.446432 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:28.446436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:28.446716 (+ 280us) server_negotiation.cc:407] Connection header received
0926 14:54:28.446808 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:28.446811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:28.446853 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:28.446909 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:28.447118 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:28.447831 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:28.448654 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:28.448886 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:28.448968 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:28.449205 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:28.449281 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:28.449339 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:28.449470 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 14:54:28.449520 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:54:28.449747 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:28.446022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.12.68.253:44649)
0926 14:54:28.446150 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:28.446154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:28.446274 (+ 120us) server_negotiation.cc:407] Connection header received
0926 14:54:28.446463 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:28.446473 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:28.446522 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:28.446586 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:28.446878 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:28.447633 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:28.448690 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:28.448995 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:28.449095 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:28.449350 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:28.449417 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:28.449473 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:28.449572 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 14:54:28.449621 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:54:28.450256 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:28.445749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.12.68.254:36075)
0926 14:54:28.446103 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:28.446107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:28.446125 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:54:28.446210 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:28.446212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:28.446355 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:28.446465 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:28.446794 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:28.447431 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:28.448574 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:28.448893 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:28.448980 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:28.449419 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:28.449751 (+ 332us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:28.449834 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:28.450071 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 14:54:28.450127 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":75,"threads_started":1}
I20260926 14:54:28.452837 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 2353
I20260926 14:54:28.458184 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 2428
I20260926 14:54:28.462827 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 2502
2026-09-26T14:54:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2971 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:28 dist-test-slave-24mn krb5kdc[2718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:28 dist-test-slave-24mn krb5kdc[2718](info): set up 2 sockets
Sep 26 14:54:28 dist-test-slave-24mn krb5kdc[2718](info): commencing operation
krb5kdc: starting...
W20260926 14:54:29.407922 12563 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/test-user.keytab.
Sep 26 14:54:29 dist-test-slave-24mn krb5kdc[2718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
Entry for principal kudu/127.12.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab.
I20260926 14:54:29.522671 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:38083
--webserver_interface=127.12.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:38083,127.12.68.253:36013,127.12.68.252:36649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:29.629765 2731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:54:29.630048 2731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:29.630085 2731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:29.630144 2731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:29.634133 2731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:29.634193 2731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:29.634212 2731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:29.634227 2731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:29.634239 2731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:29.639441 2731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:38083,127.12.68.253:36013,127.12.68.252:36649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.254.keytab
--principal=kudu/127.12.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.254:38083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:29.640468 2731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:29.641455 2731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:29.649624 2739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:29.649624 2736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:29.649746 2731 server_base.cc:1034] running on GCE node
W20260926 14:54:29.649678 2737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:29.650401 2731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:29.651453 2731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:29.652588 2731 hybrid_clock.cc:648] HybridClock initialized: now 1790434469652566 us; error 38 us; skew 500 ppm
Sep 26 14:54:29 dist-test-slave-24mn krb5kdc[2718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434469, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:29.661967 2731 init.cc:377] Logged in from keytab as kudu/127.12.68.254@KRBTEST.COM (short username kudu)
I20260926 14:54:29.662927 2731 webserver.cc:460] Webserver started at http://127.12.68.254:43275/ using document root <none> and password file <none>
I20260926 14:54:29.663466 2731 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:29.663549 2731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:29.663759 2731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:29.666030 2731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/data/instance:
uuid: "5c4a915201dd4febaad1927bf5a2fd3c"
format_stamp: "Formatted at 2026-09-26 14:54:29 on dist-test-slave-24mn"
server_key: "d44b43b161d0d6b5d8d3644a2ea1b7b1"
server_key_iv: "953d66b7d2d8d0b7e35560735a4ab38e"
server_key_version: "encryptionkey@0"
I20260926 14:54:29.666572 2731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/wal/instance:
uuid: "5c4a915201dd4febaad1927bf5a2fd3c"
format_stamp: "Formatted at 2026-09-26 14:54:29 on dist-test-slave-24mn"
server_key: "d44b43b161d0d6b5d8d3644a2ea1b7b1"
server_key_iv: "953d66b7d2d8d0b7e35560735a4ab38e"
server_key_version: "encryptionkey@0"
I20260926 14:54:29.670094 2731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:54:29.672391 2746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:29.673559 2731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:54:29.673659 2731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/wal
uuid: "5c4a915201dd4febaad1927bf5a2fd3c"
format_stamp: "Formatted at 2026-09-26 14:54:29 on dist-test-slave-24mn"
server_key: "d44b43b161d0d6b5d8d3644a2ea1b7b1"
server_key_iv: "953d66b7d2d8d0b7e35560735a4ab38e"
server_key_version: "encryptionkey@0"
I20260926 14:54:29.673753 2731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:29.694056 2731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:29.697023 2731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:29.697188 2731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:29.703532 2731 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.254:38083
I20260926 14:54:29.703787 2798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.254:38083 every 8 connection(s)
I20260926 14:54:29.704439 2731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:29.707012 2799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:29.708415 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 2731
I20260926 14:54:29.708537 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-0/wal/instance
I20260926 14:54:29.708853 12563 external_mini_cluster.cc:1428] Setting key fe61699b4bfafc9ff2f94e60048b9d9b
I20260926 14:54:29.711323 2799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.68.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.253@KRBTEST.COM" created.
Sep 26 14:54:29 dist-test-slave-24mn krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434469, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:54:29.725539 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.713652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:38083 (local address 127.0.0.1:41222)
0926 14:54:29.714171 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.714206 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:29.714983 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:29.715622 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:29.715627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:29.716079 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:29.716337 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:29.716352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.717470 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.717474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:29.718185 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:29.718194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.718818 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.718823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:29.718941 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:29.719458 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:29.719474 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:29.721011 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:29.722772 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:29.722781 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:29.722791 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:29.722997 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:29.723276 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:29.723279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:29.723281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:29.723381 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:29.723939 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:29.723945 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:29.724088 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 14:54:29.724234 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 14:54:29.724605 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":86,"thread_start_us":147,"threads_started":1}
I20260926 14:54:29.725539 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.713651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.12.68.254:38083)
0926 14:54:29.714050 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:29.714068 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:29.714303 (+ 235us) server_negotiation.cc:407] Connection header received
0926 14:54:29.715269 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:29.715274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:29.715454 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:29.715617 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:29.716468 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.717363 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.718290 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.718655 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.718809 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:29.721168 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:29.721174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:29.721183 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:29.721204 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:29.722675 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:29.723093 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:29.723099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:29.723100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:29.723195 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:29.723456 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:29.723459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:29.723461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:29.723823 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:29.723964 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:29.724304 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 14:54:29.724514 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":97,"threads_started":1}
I20260926 14:54:29.728081 2799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:54:29.728842 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.728555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:36013 (local address 127.0.0.1:39930)
0926 14:54:29.728701 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.728773 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260926 14:54:29.729074 2750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.68.253:36013: connect: Connection refused (error 111)
I20260926 14:54:29.729818 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.729624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:36013 (local address 127.0.0.1:39942)
0926 14:54:29.729722 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.729764 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:54:29.729982 2799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.253:36013: Network error: Client connection negotiation failed: client connection to 127.12.68.253:36013: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.253.keytab.
Entry for principal kudu/127.12.68.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.253.keytab.
I20260926 14:54:29.734855 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.253:36013
--webserver_interface=127.12.68.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.253.keytab
--principal=kudu/127.12.68.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:38083,127.12.68.253:36013,127.12.68.252:36649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:54:29.779217 2799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } attempt: 1
I20260926 14:54:29.780128 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.779793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:36013 (local address 127.0.0.1:39956)
0926 14:54:29.779983 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.780050 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 14:54:29.780830 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.780609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:36013 (local address 127.0.0.1:39964)
0926 14:54:29.780727 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.780774 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 14:54:29.780962 2799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.253:36013: Network error: Client connection negotiation failed: client connection to 127.12.68.253:36013: connect: Connection refused (error 111)
W20260926 14:54:29.841423 2806 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:54:29.841718 2806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:29.841763 2806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:29.841825 2806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:29.845851 2806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:29.845903 2806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:29.845929 2806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:29.845950 2806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:29.845975 2806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:29.849138 2799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } attempt: 2
I20260926 14:54:29.849901 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.849632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:36013 (local address 127.0.0.1:39978)
0926 14:54:29.849772 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.849831 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 14:54:29.850526 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.850315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:36013 (local address 127.0.0.1:39986)
0926 14:54:29.850411 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.850474 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.253:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 14:54:29.850660 2799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.253:36013: Network error: Client connection negotiation failed: client connection to 127.12.68.253:36013: connect: Connection refused (error 111)
I20260926 14:54:29.851334 2806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:38083,127.12.68.253:36013,127.12.68.252:36649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.253.keytab
--principal=kudu/127.12.68.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.253:36013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.68.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:29.852344 2806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:29.853295 2806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:29.861183 2814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:29.861207 2811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:29.861191 2812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:29.861279 2806 server_base.cc:1034] running on GCE node
I20260926 14:54:29.861969 2806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:29.862996 2806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:29.864149 2806 hybrid_clock.cc:648] HybridClock initialized: now 1790434469864134 us; error 23 us; skew 500 ppm
Sep 26 14:54:29 dist-test-slave-24mn krb5kdc[2718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434469, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:29.866662 2806 init.cc:377] Logged in from keytab as kudu/127.12.68.253@KRBTEST.COM (short username kudu)
I20260926 14:54:29.867461 2806 webserver.cc:460] Webserver started at http://127.12.68.253:43787/ using document root <none> and password file <none>
I20260926 14:54:29.867986 2806 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:29.868046 2806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:29.868221 2806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:29.870172 2806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/data/instance:
uuid: "3d897833f60f42efb512987d90c1b03a"
format_stamp: "Formatted at 2026-09-26 14:54:29 on dist-test-slave-24mn"
server_key: "186bc2e8fe26521be302e89389dda7d7"
server_key_iv: "ad0931efe58c07bfdace19ffcf4c755d"
server_key_version: "encryptionkey@0"
I20260926 14:54:29.870688 2806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/wal/instance:
uuid: "3d897833f60f42efb512987d90c1b03a"
format_stamp: "Formatted at 2026-09-26 14:54:29 on dist-test-slave-24mn"
server_key: "186bc2e8fe26521be302e89389dda7d7"
server_key_iv: "ad0931efe58c07bfdace19ffcf4c755d"
server_key_version: "encryptionkey@0"
I20260926 14:54:29.874177 2806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:54:29.876416 2821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:29.877593 2806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:54:29.877703 2806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/wal
uuid: "3d897833f60f42efb512987d90c1b03a"
format_stamp: "Formatted at 2026-09-26 14:54:29 on dist-test-slave-24mn"
server_key: "186bc2e8fe26521be302e89389dda7d7"
server_key_iv: "ad0931efe58c07bfdace19ffcf4c755d"
server_key_version: "encryptionkey@0"
I20260926 14:54:29.877802 2806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:29.887266 2806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:29.890053 2806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:29.890244 2806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:29.896253 2806 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.253:36013
I20260926 14:54:29.896524 2873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.253:36013 every 8 connection(s)
I20260926 14:54:29.897130 2806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/data/info.pb
I20260926 14:54:29.899718 2874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:29.900279 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 2806
I20260926 14:54:29.900410 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-1/wal/instance
I20260926 14:54:29.900723 12563 external_mini_cluster.cc:1428] Setting key 3241e8c2d40c7831c928c2b9a3f78dfd
I20260926 14:54:29.903863 2874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.68.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.68.252@KRBTEST.COM" created.
Sep 26 14:54:29 dist-test-slave-24mn krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434469, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.253@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:54:29.917115 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.905672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.12.68.254:38083)
0926 14:54:29.905821 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:29.905827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:29.906611 (+ 784us) server_negotiation.cc:407] Connection header received
0926 14:54:29.907510 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:29.907514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:29.907589 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:29.907693 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:29.908864 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.909434 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.910691 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.910926 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.911013 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:29.913551 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:29.913555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:29.913565 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:29.913594 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:29.915141 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:29.915604 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:29.915608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:29.915609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:29.915698 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:29.916015 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:29.916018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:29.916020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:29.916242 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:29.916339 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:29.916820 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 14:54:29.916946 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:54:29.917783 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.905996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:38083 (local address 127.0.0.1:41238)
0926 14:54:29.906450 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.906499 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:29.907308 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:29.907829 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:29.907836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:29.908194 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:29.908695 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:29.908713 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.909647 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.909651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:29.910561 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:29.910571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.911080 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.911084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:29.911272 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:29.911848 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:29.911868 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:29.913382 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:29.915256 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:29.915261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:29.915273 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:29.915508 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:29.915779 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:29.915781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:29.915783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:29.915935 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:29.916325 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:29.916330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:29.916559 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 14:54:29.916718 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 14:54:29.916957 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":121,"threads_started":1}
I20260926 14:54:29.919014 2874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:54:29 dist-test-slave-24mn krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434469, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.253@KRBTEST.COM for kudu/127.12.68.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.68.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.252.keytab.
Entry for principal kudu/127.12.68.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.252.keytab.
I20260926 14:54:29.927668 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.919456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:36013 (local address 127.0.0.1:39998)
0926 14:54:29.919556 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.919569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:29.919647 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:29.920267 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:29.920270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:29.920392 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:29.920486 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:29.920491 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.921418 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.921421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:29.921941 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:29.921948 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.922525 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.922527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:29.922607 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:29.923243 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:29.923254 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:29.924442 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:29.926064 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:29.926066 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:29.926068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:29.926301 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:29.926553 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:29.926555 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:29.926556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:29.926632 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:29.927224 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:29.927227 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:29.927301 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 14:54:29.927393 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 14:54:29.927515 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 14:54:29.927824 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.919585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39998 (local address 127.12.68.253:36013)
0926 14:54:29.919911 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:29.919921 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:29.919943 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:54:29.919976 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:29.919979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:29.920151 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:29.920288 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:29.920581 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.921323 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.922035 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.922426 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.922534 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:29.924571 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:29.924579 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:29.924589 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:29.924610 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:29.925951 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:29.926386 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:29.926391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:29.926392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:29.926468 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:29.926705 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:29.926707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:29.926708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:29.926944 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:29.927075 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:29.927486 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 14:54:29.927641 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":145,"threads_started":1}
I20260926 14:54:29.927976 12563 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
/tmp/dist-test-taskwwFvmP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.252:36649
--webserver_interface=127.12.68.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.252.keytab
--principal=kudu/127.12.68.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.68.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.68.254:38083,127.12.68.253:36013,127.12.68.252:36649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:54:29.929565 2874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:54:29.930208 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.929989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40282)
0926 14:54:29.930094 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.930156 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 14:54:29.930400 2824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
I20260926 14:54:29.931145 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.930965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40290)
0926 14:54:29.931059 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.931100 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 14:54:29.931278 2874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:36649: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
I20260926 14:54:29.941814 2799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } attempt: 3
Sep 26 14:54:29 dist-test-slave-24mn krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434469, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for kudu/127.12.68.253@KRBTEST.COM
I20260926 14:54:29.949895 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.942352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.12.68.253:36013)
0926 14:54:29.942466 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:29.942470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:29.942516 (+ 46us) server_negotiation.cc:407] Connection header received
0926 14:54:29.942651 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:29.942654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:29.942697 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:29.942760 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:29.943164 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.943686 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.944860 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.945095 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.945178 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:29.947165 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:29.947168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:29.947170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:29.947191 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:29.948423 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:29.948873 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:29.948875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:29.948877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:29.948942 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:29.949179 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:29.949181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:29.949182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:29.949334 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:29.949400 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:29.949691 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 14:54:29.949783 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:54:29.949939 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.942304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:36013 (local address 127.0.0.1:40000)
0926 14:54:29.942438 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.942453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:29.942564 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:29.942788 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:29.942791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:29.942950 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:29.943062 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:29.943068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.943781 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.943784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:29.944752 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:29.944761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:29.945193 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:29.945197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:29.945295 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:29.945756 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:29.945766 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:29.947047 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:29.948531 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:29.948535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:29.948538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:29.948785 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:29.949015 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:29.949018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:29.949020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:29.949110 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:29.949404 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:29.949407 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:29.949509 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 14:54:29.949656 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 14:54:29.949785 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 14:54:29.950970 2799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:54:29.951634 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.951410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40304)
0926 14:54:29.951515 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.951579 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 14:54:29.952301 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.952081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40314)
0926 14:54:29.952210 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.952253 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260926 14:54:29.952417 2799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:36649: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
I20260926 14:54:29.980520 2874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } attempt: 1
I20260926 14:54:29.981411 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.981069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40316)
0926 14:54:29.981262 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.981332 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 14:54:29.982232 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.982005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40320)
0926 14:54:29.982111 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.982165 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 14:54:29.982393 2874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:36649: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
I20260926 14:54:29.983573 2799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } attempt: 1
I20260926 14:54:29.984215 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.983985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40326)
0926 14:54:29.984094 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.984149 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 14:54:29.984851 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:29.984670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40328)
0926 14:54:29.984760 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:29.984802 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:54:29.984957 2799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:36649: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
W20260926 14:54:30.036926 2880 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:54:30.037200 2880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:30.037225 2880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:30.037289 2880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:30.041240 2880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:30.041289 2880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:30.041308 2880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:30.041322 2880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:30.041337 2880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:30.046422 2880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.68.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.68.254:38083,127.12.68.253:36013,127.12.68.252:36649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/krb5kdc/kudu_127.12.68.252.keytab
--principal=kudu/127.12.68.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.68.252:36649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.68.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:54:30.047411 2880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:30.048434 2880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:54:30.050565 2874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } attempt: 2
I20260926 14:54:30.051385 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.051057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40344)
0926 14:54:30.051216 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:30.051283 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 14:54:30.052078 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.051847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40348)
0926 14:54:30.051960 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:30.052012 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 14:54:30.052201 2874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:36649: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
W20260926 14:54:30.056706 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:30.056712 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:30.056691 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:30.057097 2880 server_base.cc:1034] running on GCE node
I20260926 14:54:30.057614 2880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:30.058660 2880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:30.059818 2880 hybrid_clock.cc:648] HybridClock initialized: now 1790434470059792 us; error 40 us; skew 500 ppm
I20260926 14:54:30.060089 2799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } attempt: 2
I20260926 14:54:30.060757 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.060510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40358)
0926 14:54:30.060629 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:30.060684 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 14:54:30.061400 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.061209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40372)
0926 14:54:30.061302 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:30.061345 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:54:30.061582 2799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.68.252:36649: Network error: Client connection negotiation failed: client connection to 127.12.68.252:36649: connect: Connection refused (error 111)
Sep 26 14:54:30 dist-test-slave-24mn krb5kdc[2718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434470, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:30.062529 2880 init.cc:377] Logged in from keytab as kudu/127.12.68.252@KRBTEST.COM (short username kudu)
I20260926 14:54:30.063342 2880 webserver.cc:460] Webserver started at http://127.12.68.252:33833/ using document root <none> and password file <none>
I20260926 14:54:30.063828 2880 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:30.063923 2880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:30.064105 2880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:30.066000 2880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/data/instance:
uuid: "92c280d78fff4bb2a15590cc2de23010"
format_stamp: "Formatted at 2026-09-26 14:54:30 on dist-test-slave-24mn"
server_key: "5f52fee9c65b9e6bf0c70ec3562f904f"
server_key_iv: "58139e73325b57ef8d586621d285ebd9"
server_key_version: "encryptionkey@0"
I20260926 14:54:30.066465 2880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/wal/instance:
uuid: "92c280d78fff4bb2a15590cc2de23010"
format_stamp: "Formatted at 2026-09-26 14:54:30 on dist-test-slave-24mn"
server_key: "5f52fee9c65b9e6bf0c70ec3562f904f"
server_key_iv: "58139e73325b57ef8d586621d285ebd9"
server_key_version: "encryptionkey@0"
I20260926 14:54:30.069867 2880 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 14:54:30.072057 2896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:30.073181 2880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:30.073294 2880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/data,/tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/wal
uuid: "92c280d78fff4bb2a15590cc2de23010"
format_stamp: "Formatted at 2026-09-26 14:54:30 on dist-test-slave-24mn"
server_key: "5f52fee9c65b9e6bf0c70ec3562f904f"
server_key_iv: "58139e73325b57ef8d586621d285ebd9"
server_key_version: "encryptionkey@0"
I20260926 14:54:30.073379 2880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:30.094849 2880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:30.097714 2880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:30.097877 2880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:30.104310 2880 rpc_server.cc:307] RPC server started. Bound to: 127.12.68.252:36649
I20260926 14:54:30.104573 2948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.68.252:36649 every 8 connection(s)
I20260926 14:54:30.105201 2880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/data/info.pb
I20260926 14:54:30.107774 2949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:30.111395 2949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:54:30.114064 12563 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu as pid 2880
I20260926 14:54:30.114197 12563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwwFvmP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434204424021-12563-0/minicluster-data/master-2/wal/instance
I20260926 14:54:30.114511 12563 external_mini_cluster.cc:1428] Setting key 7578d4c3ec71b441daed24e97c05ba65
Sep 26 14:54:30 dist-test-slave-24mn krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434470, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.252@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
Sep 26 14:54:30 dist-test-slave-24mn krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.252@KRBTEST.COM
I20260926 14:54:30.125303 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.113224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.12.68.254:38083)
0926 14:54:30.113398 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.113403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.114229 (+ 826us) server_negotiation.cc:407] Connection header received
0926 14:54:30.115392 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.115396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.115497 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.115551 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.116810 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.117593 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.118866 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.119150 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.119248 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.121828 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:30.121831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:30.121833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:30.121856 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:30.123270 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.123768 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.123771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.123773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.123857 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.124238 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.124240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.124241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.124465 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:30.124548 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:30.125040 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 14:54:30.125147 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 14:54:30.126135 2951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.113575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.254:38083 (local address 127.0.0.1:41250)
0926 14:54:30.114071 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:30.114126 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:30.115158 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:30.115784 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:30.115790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:30.116234 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:30.116631 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:30.116649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.117712 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.117715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:30.118749 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:30.118760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.119279 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.119282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:30.119451 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.120027 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:30.120047 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:30.121687 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:30.123368 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:30.123375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:30.123382 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:30.123673 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:30.123959 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:30.123963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:30.123965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:30.124153 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:30.124557 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:30.124570 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:30.124814 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 14:54:30.124964 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 14:54:30.125260 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":28,"thread_start_us":115,"threads_started":1}
I20260926 14:54:30.126134 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.115141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.12.68.252:36649)
0926 14:54:30.115455 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.115464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.115497 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:54:30.115645 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.115658 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.115821 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.115992 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.116805 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.117947 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.118989 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.119456 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.119530 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.122217 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:30.122222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:30.122233 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:30.122257 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:30.123827 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.124321 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.124326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.124328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.124403 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.124643 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.124645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.124646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.124868 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:30.125036 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:30.125248 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 14:54:30.125593 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":225,"thread_start_us":128,"threads_started":1}
I20260926 14:54:30.127391 2949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:54:30 dist-test-slave-24mn krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434470, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.252@KRBTEST.COM for kudu/127.12.68.253@KRBTEST.COM
Sep 26 14:54:30 dist-test-slave-24mn krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.254@KRBTEST.COM
I20260926 14:54:30.135239 2951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.127832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.253:36013 (local address 127.0.0.1:40006)
0926 14:54:30.127997 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:30.128011 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:30.128080 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:30.128393 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:30.128396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:30.128561 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:30.128664 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:30.128671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.129425 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.129428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:30.130049 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:30.130060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.130455 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.130459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:30.130559 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.131000 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:30.131015 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:30.132259 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:30.133945 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:30.133949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:30.133951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:30.134137 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:30.134377 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:30.134380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:30.134382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:30.134457 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:30.134791 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:30.134795 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:30.134878 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 14:54:30.134958 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 14:54:30.135083 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 14:54:30.135242 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.127958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40006 (local address 127.12.68.253:36013)
0926 14:54:30.128065 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.128068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.128079 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:54:30.128157 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.128160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.128208 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.128282 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.128778 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.129326 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.130164 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.130370 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.130436 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.132400 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:30.132403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:30.132405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:30.132432 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:30.133843 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.134220 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.134224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.134226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.134300 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.134531 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.134535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.134536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.134708 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:30.134788 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:30.135015 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 14:54:30.135114 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:54:30.135962 2949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:54:30.136091 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.128242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.12.68.254:38083)
0926 14:54:30.128337 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.128340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.128452 (+ 112us) server_negotiation.cc:407] Connection header received
0926 14:54:30.128549 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.128551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.128588 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.128642 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.129182 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.129702 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.130566 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.130806 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.130901 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.133122 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:30.133126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:30.133128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:30.133153 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:30.134307 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.134687 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.134690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.134692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.134769 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.135011 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.135014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.135016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.135181 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:30.135762 (+ 581us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:30.135825 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 14:54:30.135958 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 14:54:30 dist-test-slave-24mn krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434470, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.252@KRBTEST.COM for kudu/127.12.68.252@KRBTEST.COM
Sep 26 14:54:30 dist-test-slave-24mn krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.68.253@KRBTEST.COM
I20260926 14:54:30.143327 2874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } attempt: 3
I20260926 14:54:30.143757 2951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.136385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40402)
0926 14:54:30.136469 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:30.136480 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:30.136539 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:30.136751 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:30.136754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:30.136871 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:30.136953 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:30.136959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.137885 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.137888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:30.138632 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:30.138642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.139172 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.139175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:30.139229 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.139616 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:30.139626 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:30.140988 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:30.142394 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:30.142397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:30.142399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:30.142613 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:30.142858 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:30.142861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:30.142863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:30.142961 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:30.143302 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:30.143306 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:30.143395 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 14:54:30.143497 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:54:30.143606 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 14:54:30.144191 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.136500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40402 (local address 127.12.68.252:36649)
0926 14:54:30.136597 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.136600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.136611 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:54:30.136637 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.136640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.136672 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.136731 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.137056 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.137785 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.138741 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.139069 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.139167 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.141105 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:30.141108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:30.141110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:30.141130 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:30.142302 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.142696 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.142699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.142701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.142778 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.143038 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.143040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.143042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.143217 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:30.143286 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:30.143961 (+ 675us) server_negotiation.cc:299] Negotiation successful
0926 14:54:30.144060 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:54:30.145339 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.137131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.12.68.253:36013)
0926 14:54:30.137224 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.137227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.137240 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:54:30.137331 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.137333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.137364 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.137419 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.138038 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.138749 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.139680 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.140028 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.140125 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.142314 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:30.142318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:30.142320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:30.142344 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:30.143801 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.144267 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.144270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.144272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.144346 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.144620 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.144623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.144625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.144818 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:30.145034 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:30.145115 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 14:54:30.145206 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:54:30.147231 2949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010: Bootstrap starting.
Sep 26 14:54:30 dist-test-slave-24mn krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434469, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.253@KRBTEST.COM for kudu/127.12.68.252@KRBTEST.COM
I20260926 14:54:30.149704 2949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:30.150578 2949 log.cc:824] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:30.151310 12563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:54:30.152055 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.143670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40412)
0926 14:54:30.143758 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:30.143771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:30.143840 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:30.144343 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:30.144346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:30.144502 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:30.144607 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:30.144613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.145932 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.145935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:30.146430 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:30.146437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.146950 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.146952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:30.147008 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.147386 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:30.147399 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:30.148649 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:30.150499 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:30.150502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:30.150504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:30.150766 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:30.151050 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:30.151053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:30.151054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:30.151141 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:30.151586 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:30.151589 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:30.151676 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 14:54:30.151757 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 14:54:30.151905 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 14:54:30.152094 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.143817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.12.68.252:36649)
0926 14:54:30.144130 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.144140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.144153 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:54:30.144186 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.144189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.144226 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.144705 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.144769 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.145817 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.146539 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.146855 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.146952 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.148781 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:30.148785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:30.148787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:30.148810 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:30.150322 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.150853 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.150856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.150858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.150938 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.151255 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.151257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.151259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.151496 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:30.151584 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:30.151815 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 14:54:30.151936 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":125,"threads_started":1}
I20260926 14:54:30.152807 2949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010: No bootstrap required, opened a new log
I20260926 14:54:30.152700 2773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5c4a915201dd4febaad1927bf5a2fd3c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41256
I20260926 14:54:30.155085 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a: Bootstrap starting.
I20260926 14:54:30.155725 2949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } }
I20260926 14:54:30.155978 2949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c280d78fff4bb2a15590cc2de23010, State: Initialized, Role: FOLLOWER
I20260926 14:54:30.156250 2949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } }
I20260926 14:54:30.156662 2956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } } }
I20260926 14:54:30.156777 2956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:54:30.157377 2949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:30.157596 2874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:30.158370 2874 log.cc:824] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:30.160708 2799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } attempt: 3
I20260926 14:54:30.160907 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a: No bootstrap required, opened a new log
W20260926 14:54:30.162108 2968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:54:30.162182 2968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:54:30.160648 2880 master_runner.cc:186] Error getting master registration for 127.12.68.254:38083: OK, instance_id { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" instance_seqno: 1790434469701844 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:54:30.164322 2880 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:54:30.165314 2874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } }
I20260926 14:54:30.165566 2874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d897833f60f42efb512987d90c1b03a, State: Initialized, Role: FOLLOWER
I20260926 14:54:30.165930 2874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } }
Sep 26 14:54:30 dist-test-slave-24mn krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434469, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.68.254@KRBTEST.COM for kudu/127.12.68.252@KRBTEST.COM
I20260926 14:54:30.166587 2970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } } }
I20260926 14:54:30.166796 2970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:54:30.167428 2874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:30.169840 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.161034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.68.252:36649 (local address 127.0.0.1:40416)
0926 14:54:30.161146 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:30.161162 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:30.161246 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:30.161638 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:30.161641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:30.161824 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:30.161922 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:30.161928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.162972 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.162975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:30.163774 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:30.163784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.164482 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.164486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:30.164579 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.165053 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:30.165068 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:30.166484 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:30.168378 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:30.168381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:30.168383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:30.168624 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:30.168895 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:30.168897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:30.168899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:30.168992 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:30.169356 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:30.169360 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:30.169454 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 14:54:30.169551 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 14:54:30.169679 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 14:54:30.169840 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.161233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.12.68.252:36649)
0926 14:54:30.161334 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.161339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.161352 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:54:30.161390 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.161393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.161437 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.161525 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.162044 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.162849 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.163929 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.164268 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.164366 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.166633 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:30.166637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:30.166639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:30.166663 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:30.168262 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.168714 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.168718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.168720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.168813 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.169078 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.169081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.169083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.169278 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:30.169364 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:30.169611 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 14:54:30.169713 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:54:30.173223 2799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c: Bootstrap starting.
W20260926 14:54:30.178876 2981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:54:30.179168 2981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:54:30.175864 2799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:30.175915 2806 master_runner.cc:186] Error getting master registration for 127.12.68.254:38083: OK, instance_id { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" instance_seqno: 1790434469701844 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:54:30.179771 2806 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:54:30.180512 2799 log.cc:824] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:30.182862 2799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c: No bootstrap required, opened a new log
I20260926 14:54:30.185866 2799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } }
I20260926 14:54:30.186087 2799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4a915201dd4febaad1927bf5a2fd3c, State: Initialized, Role: FOLLOWER
I20260926 14:54:30.186421 2799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } }
I20260926 14:54:30.186985 2984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } } }
I20260926 14:54:30.187256 2984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:54:30.187773 2799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:30.187860 2773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 14:54:30.188032 2773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:30.189322 2773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } }
I20260926 14:54:30.190222 2773 leader_election.cc:288] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [CANDIDATE]: Term 1 election: Requested vote from peers 3d897833f60f42efb512987d90c1b03a (127.12.68.253:36013), 92c280d78fff4bb2a15590cc2de23010 (127.12.68.252:36649)
I20260926 14:54:30.192394 2848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3d897833f60f42efb512987d90c1b03a"
I20260926 14:54:30.192625 2848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:30.193825 2848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c4a915201dd4febaad1927bf5a2fd3c in term 1.
I20260926 14:54:30.194248 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "92c280d78fff4bb2a15590cc2de23010"
I20260926 14:54:30.194469 2923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:30.195676 2923 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c4a915201dd4febaad1927bf5a2fd3c in term 1.
I20260926 14:54:30.196527 2749 leader_election.cc:302] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c4a915201dd4febaad1927bf5a2fd3c, 92c280d78fff4bb2a15590cc2de23010; no voters:
W20260926 14:54:30.198678 2752 tablet.cc:2369] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:54:30.199685 2731 master_runner.cc:263] No leader master found from master 5c4a915201dd4febaad1927bf5a2fd3c
I20260926 14:54:30.199748 2731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 14:54:30.201738 2995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:54:30.201867 2995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:54:30.202653 2984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:30.202854 2984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [term 1 LEADER]: Becoming Leader. State: Replica: 5c4a915201dd4febaad1927bf5a2fd3c, State: Running, Role: LEADER
I20260926 14:54:30.203167 2984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } }
I20260926 14:54:30.212539 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.197778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.12.68.253:36013)
0926 14:54:30.197873 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.197877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.204184 (+ 6307us) server_negotiation.cc:407] Connection header received
0926 14:54:30.204265 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.204268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.204316 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.204381 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.205213 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.205929 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.206998 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.207339 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.207434 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.208989 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:30.208992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:30.208995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:30.209021 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:30.210528 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.210982 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.210986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.210988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.211065 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.211331 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.211334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.211336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.211559 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:30.211688 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:30.212271 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 14:54:30.212378 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 14:54:30.214672 2984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c4a915201dd4febaad1927bf5a2fd3c. Latest consensus state: current_term: 1 leader_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } } }
I20260926 14:54:30.214802 2984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:30.216358 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.197203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.12.68.254:38083)
0926 14:54:30.197305 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.197309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.198284 (+ 975us) server_negotiation.cc:407] Connection header received
0926 14:54:30.198402 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.198405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.198451 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.198519 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.200674 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.201439 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.202600 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.202942 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.203039 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.205630 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:30.205634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:30.205637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:30.205661 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:30.207264 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.215033 (+ 7769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.215036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.215038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.215124 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.215409 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.215412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.215414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.215630 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:30.215750 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:30.216092 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 14:54:30.216210 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:54:30.220131 3006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:30.221093 3006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:30.225658 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:30.197158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.12.68.252:36649)
0926 14:54:30.197265 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.197270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.204146 (+ 6876us) server_negotiation.cc:407] Connection header received
0926 14:54:30.208003 (+ 3857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.208007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.208071 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.208122 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.208932 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.209686 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.210814 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:30.211159 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:30.211233 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:30.212851 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:30.212855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:30.212857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:30.212884 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:30.214462 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.224011 (+ 9549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.224016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.224018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.224145 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:30.224496 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:30.224499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:30.224501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:30.224852 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:30.224990 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:30.225350 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 14:54:30.225485 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:54:30.226253 2923 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010 [term 1 FOLLOWER]: Refusing update from remote peer 5c4a915201dd4febaad1927bf5a2fd3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:30.226913 3004 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:30.230856 2956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c4a915201dd4febaad1927bf5a2fd3c. Latest consensus state: current_term: 1 leader_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } } }
I20260926 14:54:30.230962 2956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:54:30.232224 2847 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a [term 1 FOLLOWER]: Refusing update from remote peer 5c4a915201dd4febaad1927bf5a2fd3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:30.232759 3004 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:30.238111 3004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } } }
I20260926 14:54:30.238229 3004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:30.239377 2956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } } }
I20260926 14:54:30.239481 2956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:54:30.239825 3007 mvcc.cc:205] Tried to move back new op lower bound from 7333619590039969792 to 7333619589971054592. Current Snapshot: MvccSnapshot[applied={T|T < 7333619590039969792}]
I20260926 14:54:30.240769 2970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c4a915201dd4febaad1927bf5a2fd3c. Latest consensus state: current_term: 1 leader_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } } }
I20260926 14:54:30.240890 2970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:54:30.244542 3006 catalog_manager.cc:1347] Generated new cluster ID: 2f3421a3282442298c0550bd4b96d9ad
I20260926 14:54:30.244616 3006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:30.263981 2984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } } }
I20260926 14:54:30.264114 2984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:30.265230 2970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4a915201dd4febaad1927bf5a2fd3c" member_type: VOTER last_known_addr { host: "127.12.68.254" port: 38083 } } peers { permanent_uuid: "3d897833f60f42efb512987d90c1b03a" member_type: VOTER last_known_addr { host: "127.12.68.253" port: 36013 } } peers { permanent_uuid: "92c280d78fff4bb2a15590cc2de23010" member_type: VOTER last_known_addr { host: "127.12.68.252" port: 36649 } } }
I20260926 14:54:30.265344 2970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:54:30.270998 3006 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:30.271971 3006 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:30.295797 3006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c4a915201dd4febaad1927bf5a2fd3c: Generated new TSK 0
I20260926 14:54:30.296710 3006 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:54:30.327435 2879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:30.324918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.12.68.253:36013)
0926 14:54:30.325042 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.325046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.326333 (+ 1287us) server_negotiation.cc:407] Connection header received
0926 14:54:30.326534 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.326537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.326619 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.326676 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.327307 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40046: BlockingRecv error: recv got EOF from 127.0.0.1:40046 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 14:54:30.328890 2954 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:30.325693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.12.68.252:36649)
0926 14:54:30.325804 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.325808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.327947 (+ 2139us) server_negotiation.cc:407] Connection header received
0926 14:54:30.328192 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.328196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.328245 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.328292 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.328771 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40436: BlockingRecv error: recv got EOF from 127.0.0.1:40436 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 14:54:30.330803 2802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:30.329473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.12.68.254:38083)
0926 14:54:30.329595 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.329600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.329929 (+ 329us) server_negotiation.cc:407] Connection header received
0926 14:54:30.330088 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.330091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.330140 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.330195 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.330687 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41274: BlockingRecv error: recv got EOF from 127.0.0.1:41274 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 14:54:30.335285 2879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:30.333695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40060 (local address 127.12.68.253:36013)
0926 14:54:30.333803 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.333807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.334288 (+ 481us) server_negotiation.cc:407] Connection header received
0926 14:54:30.334516 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.334520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.334611 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.334668 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.335182 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40060: BlockingRecv error: recv got EOF from 127.0.0.1:40060 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 14:54:30.340138 2879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:30.338583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40064 (local address 127.12.68.253:36013)
0926 14:54:30.338687 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.338690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.339213 (+ 523us) server_negotiation.cc:407] Connection header received
0926 14:54:30.339402 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.339405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.339456 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.339506 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.340040 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40064: BlockingRecv error: recv got EOF from 127.0.0.1:40064 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 14:54:30.345679 2879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:30.344140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.12.68.253:36013)
0926 14:54:30.344239 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.344243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.344775 (+ 532us) server_negotiation.cc:407] Connection header received
0926 14:54:30.344965 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.344968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.345018 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.345067 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.345580 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40076: BlockingRecv error: recv got EOF from 127.0.0.1:40076 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 14:54:30.353097 2879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:30.351644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.12.68.253:36013)
0926 14:54:30.351765 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.351768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.352276 (+ 508us) server_negotiation.cc:407] Connection header received
0926 14:54:30.352441 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.352444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.352489 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.352533 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.352999 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40086: BlockingRecv error: recv got EOF from 127.0.0.1:40086 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 14:54:30.364936 2879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:30.363209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.12.68.253:36013)
0926 14:54:30.363330 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.363335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.363929 (+ 594us) server_negotiation.cc:407] Connection header received
0926 14:54:30.364148 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.364152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.364211 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.364266 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.364828 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40100: BlockingRecv error: recv got EOF from 127.0.0.1:40100 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 14:54:30.384023 2879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:30.382908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.12.68.253:36013)
0926 14:54:30.383018 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.383022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.383241 (+ 219us) server_negotiation.cc:407] Connection header received
0926 14:54:30.383365 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.383368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.383416 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.383480 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.383912 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40108: BlockingRecv error: recv got EOF from 127.0.0.1:40108 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 14:54:30.419092 2879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:30.418145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.12.68.253:36013)
0926 14:54:30.418288 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.418292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.418376 (+ 84us) server_negotiation.cc:407] Connection header received
0926 14:54:30.418485 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.418487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.418540 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.418619 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.418986 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40110: BlockingRecv error: recv got EOF from 127.0.0.1:40110 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 14:54:30.486083 2879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:30.485065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.12.68.253:36013)
0926 14:54:30.485189 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.485193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.485313 (+ 120us) server_negotiation.cc:407] Connection header received
0926 14:54:30.485430 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.485432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.485484 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.485564 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.485978 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40120: BlockingRecv error: recv got EOF from 127.0.0.1:40120 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 14:54:30.617249 2879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:30.616030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.12.68.253:36013)
0926 14:54:30.616188 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.616193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.616337 (+ 144us) server_negotiation.cc:407] Connection header received
0926 14:54:30.616435 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.616437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.616513 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.616595 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.617129 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40124: BlockingRecv error: recv got EOF from 127.0.0.1:40124 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 14:54:30.876180 2879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:30.875138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.12.68.253:36013)
0926 14:54:30.875274 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:30.875278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:30.875328 (+ 50us) server_negotiation.cc:407] Connection header received
0926 14:54:30.875492 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:30.875494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:30.875541 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:30.875612 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:30.876068 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40126: BlockingRecv error: recv got EOF from 127.0.0.1:40126 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:54:31.163494 2968 catalog_manager.cc:1259] Loaded cluster ID: 2f3421a3282442298c0550bd4b96d9ad
I20260926 14:54:31.163582 2968 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010: loading cluster ID for follower catalog manager: success
I20260926 14:54:31.165266 2968 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010: acquiring CA information for follower catalog manager: success
I20260926 14:54:31.166386 2968 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92c280d78fff4bb2a15590cc2de23010: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:54:31.180410 2981 catalog_manager.cc:1259] Loaded cluster ID: 2f3421a3282442298c0550bd4b96d9ad
I20260926 14:54:31.180478 2981 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a: loading cluster ID for follower catalog manager: success
I20260926 14:54:31.182245 2981 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a: acquiring CA information for follower catalog manager: success
I20260926 14:54:31.183350 2981 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d897833f60f42efb512987d90c1b03a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:54:31.394228 3080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:31.390406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40136 (local address 127.12.68.253:36013)
0926 14:54:31.390669 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:31.390673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:31.390781 (+ 108us) server_negotiation.cc:407] Connection header received
0926 14:54:31.390935 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:31.390938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:31.391065 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:31.391158 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:31.391447 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:31.392023 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:31.392868 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:31.393096 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:31.393190 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:31.393503 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:31.393922 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:31.394016 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:31.394062 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 14:54:31.394111 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260926 14:54:31.400415 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:31.397064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40440 (local address 127.12.68.252:36649)
0926 14:54:31.397271 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:31.397274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:31.397286 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:54:31.397356 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:31.397359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:31.397463 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:31.397556 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:31.397807 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:31.398326 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:31.399145 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:31.399361 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:31.399436 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:31.399647 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:31.400091 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:31.400179 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:31.400222 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 14:54:31.400269 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":59,"threads_started":1}
I20260926 14:54:31.406528 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:31.403177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.12.68.252:36649)
0926 14:54:31.403296 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:31.403299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:31.403454 (+ 155us) server_negotiation.cc:407] Connection header received
0926 14:54:31.403777 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:31.403780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:31.403823 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:31.403990 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:31.404198 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:31.404697 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:31.405629 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:31.405841 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:31.405923 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:31.406171 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:31.406231 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:31.406284 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:31.406366 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 14:54:31.406409 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:54:31.407162 3080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:31.402903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.12.68.253:36013)
0926 14:54:31.403008 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:31.403012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:31.403423 (+ 411us) server_negotiation.cc:407] Connection header received
0926 14:54:31.403538 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:31.403541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:31.403600 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:31.403671 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:31.404031 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:31.404834 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:31.406107 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:31.406356 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:31.406444 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:31.406713 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:31.406786 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:31.406831 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:31.406992 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 14:54:31.407044 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:54:31.408025 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:31.402887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.12.68.254:38083)
0926 14:54:31.403400 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:31.403405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:31.403417 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:54:31.403503 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:31.403506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:31.403655 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:31.403761 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:31.404057 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:31.404844 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:31.406610 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:31.406938 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:31.407032 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:31.407408 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:31.407644 (+ 236us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:31.407719 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:31.407809 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 14:54:31.407866 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":67,"threads_started":1}
I20260926 14:54:31.410888 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 2731
I20260926 14:54:31.416303 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 2806
I20260926 14:54:31.421092 12563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwwFvmP/build/debug/bin/kudu with pid 2880
2026-09-26T14:54:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2957 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5929 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (266990 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST